Display apparatus and method for controlling display apparatus

ABSTRACT

An HMD includes a six-axis sensor, a magnetic sensor, and a head motion detecting section that detect at least one of the position and motion of a head, a reference setting section that sets a reference state based on at least one of the head position and motion detected by the head motion detecting section, and a display controlling section that changes the display state of a content displayed by the display section based on changes in the position and motion of the head with respect to those in the reference state. The content is formed of a plurality of contents, and the display controlling section changes the display state of the contents displayed by an image display section in such a way that the relative display positions of the contents are maintained.

BACKGROUND 1. Technical Field

The present invention relates to a display apparatus and a method forcontrolling the display apparatus.

2. Related Art

There is a known display apparatus that includes a display section wornon a user's head and allows the user to visually recognize an image (seeJP-A-2016-149002, for example).

JP-A-2016-149002 discloses an apparatus that produces a plurality ofvirtual displays arranged in a three-dimensional virtual space, selectsone of the virtual displays in accordance with the head's motiondetected with a detector, and reproduces a content related to theselected virtual display.

The apparatus disclosed in JP-A-2016-149002 produces the plurality ofvirtual displays in such a way that they are arranged in thethree-dimensional virtual space. When the number of arranged virtualdisplays increases, it is difficult in some cases for the user to selecta virtual display on which a content that the user desires to view isdisplayed. Further, in a case where the plurality of virtual displaysare arranged in front of the user, contents displayed in front of theuser could undesirably block the user's visual field and hence hamperthe user's operation. Moreover, in the case of a display apparatus wornon the user's head, since a displayed image lowers the visibility of anoutside scene in some cases, it is difficult in some cases for the userto operate an operation section of the display apparatus.

SUMMARY

An advantage of some aspects of the invention is to allow a user toreadily select a content to be displayed and change the display state ofthe content through simple operation.

An aspect of the invention is directed to a display apparatus includinga display section that is worn on a user's head, a detection sectionthat detects at least one of a position and a motion of a detectiontarget section including the head, a reference setting section that setsa reference state based on at least one of the position and motion ofthe detection target section detected by the detection section, and adisplay controlling section that changes display states of contentsdisplayed by the display section based on changes in the position andmotion of the detection target section with respect to the position andmotion of the detection target section in the reference state, and thedisplay controlling section changes the display states of the contentsdisplayed by the display section in such a way that relative displaypositions of a plurality of the contents are maintained.

According to the configuration described above, the reference state canbe set on the basis of at least one of the position and motion of thedetection target section, and the display states of the plurality ofcontents can be changed on the basis of changes in the position andmotion of the detection target section with respect to those in the setreference state. Further, when the display states of the contents arechanged, they can be so changed that the relative display positions ofthe plurality of contents are maintained. Therefore, a content to bedisplayed can be readily selected, and the display state of the contentcan be changed through simple operation.

In the aspect of the invention, the display apparatus may furtherinclude a storage section that stores display position informationrepresenting the relative display positions of a plurality of thecontents, and the display controlling section may change the displaystates of the contents in such a way that the relative display positionsof a plurality of the contents represented by the display positioninformation are maintained.

According to the aspect of the invention with this configuration, thedisplay states of the contents displayed by the display section can beso changed that the relative display positions of the plurality ofcontents are maintained.

In the aspect of the invention, the display controlling section mayselect one of the contents based on changes in the position and motionof the detection target section with respect to the position and motionof the detection target section in the reference state and cause thedisplay section to display the selected content.

According to the aspect of the invention with this configuration, acontent can be selected on the basis of the amounts of changes in theposition and motion of the detection target section, and the selectedcontent can be displayed. A content can therefore be selected by theposition or motion of the head, whereby a content can be readilyselected.

In the aspect of the invention, the reference setting section may set,as the reference state, an axis of rotation of the detection targetsection and the position of the detection target section that rotatesaround the axis of rotation, and the display controlling section may becapable of setting the display positions of the contents along arotational direction around the axis of rotation and along a directionperpendicular to the rotational direction.

According to the aspect of the invention with this configuration, theplurality of contents can be so set that the relative display positionsof the plurality of contents are maintained.

In the aspect of the invention, the display controlling section may becapable of setting the display position of any of the contents on a rearside of the detection target section in the reference state.

According to the aspect of the invention with this configuration, thedisplay positions of the contents can be changed in a variety ofmanners.

In the aspect of the invention, the display controlling section may becapable of setting the display positions of the contents in positionswhere the user is allowed to visually recognize a plurality of thecontents arranged on a spherical surface around a point on the axis ofrotation or in a plane separate from the axis of rotation by apredetermined distance.

According to the aspect of the invention with this configuration, theplurality of contents are set in positions where the user can visuallyrecognize the contents arranged on a spherical surface around a point onthe axis of rotation or in a plane separate from the axis of rotation bya predetermined distance. The display states of the plurality ofcontents can therefore be so changed that the relative display positionsof the contents are maintained.

In the aspect of the invention, the display controlling section may becapable of setting the display position of any of the contents in aposition where the user is allowed to visually recognize the contentdisposed in a plane containing the user's feet.

According to the aspect of the invention with this configuration, theuser can visually recognize the content disposed in the plane containingthe user's feet.

In the aspect of the invention, the display section may include a rightdisplay section that allows the user's right eye to visually recognizean image and a left display section that allows the user's left eye tovisually recognize an image, and the display controlling section maycause any of the contents to be displayed with parallax present betweenthe right display section and the left display section.

According to the aspect of the invention with this configuration, any ofthe contents can be displayed with parallax present between the rightdisplay section and the left display section. The content can thereforebe displayed in a three-dimensional manner.

In the aspect of the invention, the reference setting section may setthe reference state in accordance with a position or a motion of apointing element.

According to the aspect of the invention with this configuration, thereference state can be set in accordance with the position or motion ofthe pointing element. The reference state can therefore be readily set.

In the aspect of the invention, the display controlling section may seta display magnification of any of the contents in the display section inaccordance with the position or motion of the pointing element.

According to the aspect of the invention with this configuration, thedisplay magnification of any of the contents can be set in accordancewith the position or motion of the pointing element. The displaymagnification of the content can therefore be set through simpleoperation.

In the aspect of the invention, the display controlling section maycause the display section to display a content for input used for inputoperation performed on any of the contents.

According to the aspect of the invention with this configuration, thedisplay section displays the content for input. Input operation cantherefore be performed on the content via the content for input.

In the aspect of the invention, the reference setting section may set,when the reference setting section determines that the detection targetsection has been stationary for at least a predetermined period, theposition of the stationary detection target section as the referencestate.

According to the aspect of the invention with this configuration, whenit is determined that the detection target section has been stationaryfor at least a predetermined period, the position of the stationarydetection target section is set as the reference state. The referencestate will therefore not be set in a position where the user does notintend.

In the aspect of the invention, the display apparatus may furtherinclude a sightline detection section that detects orientations of theuser's sightlines, and the reference setting section may set thereference state including the position or motion of the detection targetsection and the orientations of the sightlines detected by the sightlinedetection section.

According to the aspect of the invention with this configuration, thereference state can be set on the basis of the position or motion of thedetection target section and the orientations of the sightlines detectedby the sightline detection section. The reference state can therefore beset in more detail in correspondence with the user's state.

In the aspect of the invention, the detection section may be disposed inan enclosure of the display section and include at least any of a GPSdetection section that performs position detection based on GPS signals,an acceleration sensor that detects acceleration, an angularacceleration sensor that detects angular acceleration, a monocularcamera, a stereocamera, and a magnetic sensor.

According to the aspect of the invention with this configuration, thedetection section can detect the position or motion of the detectiontarget section with precision.

In the aspect of the invention, the magnetic sensor may be disposed on aside facing an outer surface of the enclosure of the display section.

According to the aspect of the invention with this configuration, theinfluence of the enclosure on the magnetism detection performed by themagnetic sensor can be reduced.

In the aspect of the invention, the display apparatus may furtherinclude a processing section that performs a plurality of functions, andeach of the contents corresponds to the functions performed by theprocessing section.

According to the aspect of the invention with this configuration, acontent corresponding to the functions performed by the processingsection can be displayed by the display section.

In the aspect of the invention, the display apparatus may furtherinclude an editing processing section that changes a substance of any ofthe contents displayed by the display section.

According to the aspect of the invention with this configuration, theediting processing section can change the substance of any of thecontents.

In the aspect of the invention, the display controlling section maycause a plurality of the contents displayed by the display section to bedisplayed in a form of thumbnail images based on the amounts of changesin the position and motion of the detection target section with respectto the amounts of changes in the position and motion of the detectiontarget section in the reference state.

According to the aspect of the invention with this configuration, theplurality of contents displayed by the display section can be displayedin the form of thumbnail images. The user can therefore readily graspthe contents displayed by the display section.

In the aspect of the invention, the display controlling section maypredict changes in the position and motion of the detection targetsection with respect to the position and motion of the detection targetsection in the reference state and change the display state of any ofthe contents in accordance with a result of the prediction.

According to the aspect of the invention with this configuration, thedisplay state of any of the contents can be changed in accordance with aresult of the prediction of changes in the position and motion of thedetection target section. The contents displayed by the display sectioncan therefore be displayed in a plurality of positions. The displaystate of any of the contents can therefore be changed at an earlytiming.

In the aspect of the invention, the display controlling section maycause the display section to display a plurality of the contents in sucha way that the plurality of contents are arranged in different positionsin a direction extending from a near side toward a far side with respectto the detection target section.

According to the aspect of the invention with this configuration, theplurality of contents are arranged in different positions in thedirection extending from the near side toward the far side with respectto the detection target section. The contents displayed by the displaysection can therefore be displayed in a plurality of positions.

Another aspect of the invention is directed to a method for controllinga display apparatus including a display section that is worn on a user'shead, the method including a detecting step of detecting at least one ofa position and a motion of a detection target section including thehead, a setting step of setting a reference state based on at least oneof the position and motion of the detection target section detected inthe detecting step, and a changing step of changing display states ofcontents displayed by the display section based on changes in theposition and motion of the detection target section with respect to theposition and motion of the detection target section in the referencestate, and the display states of the contents displayed by the displaysection are so changed that relative display positions of a plurality ofthe contents are maintained.

According to the configuration described above, the reference state canbe set on the basis of at least one of the position and motion of thedetection target section, and the display states of the contents can bechanged on the basis of changes in the position and motion of thedetection target section with respect to those in the set referencestate. Further, when the display states of the contents are changed,they can be so changed that the relative display positions of thecontents are maintained. Therefore, a content to be displayed can bereadily selected, and the display state of the content can be changedthrough simple operation.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will be described with reference to the accompanyingdrawings, wherein like numbers reference like elements.

FIG. 1 is an exterior view of an HMD.

FIG. 2 is a key part plan view showing the configuration of an opticalsystem of the HMD.

FIG. 3 is a perspective view showing the configuration of an imagedisplay section.

FIG. 4 is a block diagram of the HMD.

FIG. 5 is a functional block diagram of a control apparatus.

FIG. 6 is a block diagram showing the configuration of an operationdevice.

FIG. 7 shows a visual field visually recognized by a user through theimage display section.

FIG. 8 shows a content related to coordinates in a spherical coordinatesystem.

FIG. 9 is a top view showing the user and contents set at coordinates inthe spherical coordinate system.

FIG. 10 is a top view showing the user and the contents set at thecoordinates in the spherical coordinate system.

FIG. 11 shows the user and contents set at coordinates in the sphericalcoordinate system.

FIG. 12 shows the user and contents set at coordinates in the sphericalcoordinate system.

FIG. 13 shows that input operation is performed on the basis ofsightlines detected with sightline sensors.

FIG. 14 shows an example of an image displayed at the user's feet.

FIG. 15 shows an image for selecting an application or an applicationscreen.

FIG. 16 is a flowchart showing the action of the HMD.

FIG. 17 is a flowchart showing the action of the HMD.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

FIG. 1 is an exterior view showing an exterior configuration of an HMD(head mounted display) 100, which is configured by using a displayapparatus and a method for controlling the display apparatus accordingto an embodiment of the invention.

The HMD 100 is a display apparatus including an image display section 20(display section), which is worn on a user's head and allows the user tovisually recognize a virtual image, and a control apparatus 10, whichcontrols the image display section 20. A person who wears the imagedisplay section 20 on the head is called a user.

The control apparatus 10 includes a flat, box-shaped case 10A (can alsobe called enclosure or main body), as shown in FIG. 1. The case 10Aincludes buttons 11, an LED indicator 12, a trackpad 14, up/down keys15, a changeover switch 16, and a power switch 18. The buttons 11, thetrackpad 14, the up/down keys 15, the changeover switch 16, and thepower switch 18 are operation sections operated by the user. The LEDindicator 12 functions as a secondary display section showing, forexample, the action state of the HMD 100. The user can operate the HMD100 by operating the operation sections. The control apparatus 10functions as a controller of the HMD 100.

The image display section 20 is a wearable part worn on the user's headand has a glasses-like shape in the present embodiment. The imagedisplay section 20 includes a main body formed of a right holder 21, aleft holder 23, and a front frame 27, and the main body includes a rightdisplay unit 22, a left display unit 24, a right light guide plate 26,and a left light guide plate 28.

The right holder 21 and the left holder 23, which extend rearward fromthe opposite ends of the front frame 27, hold the image display section20 on the user's head, as the temples (bows) of spectacles do. One ofthe opposite ends of the front frame 27 or the end located on the rightof the user who wears the image display apparatus 20 is called an endER, and the other end or the end located on the left of the user iscalled an end EL. The right holder 21 is so provided as to extend fromthe end ER of the front frame 27 to a position corresponding to a righttemporal region of the user who wears the image display section 20. Theleft holder 23 is so provided as to extend from the end EL of the frontframe 27 to a position corresponding to a left temporal region of theuser who wears the image display section 20.

The right light guide plate 26 and the left light guide plate 28 areprovided as part of the front frame 27. The right light guide plate 26is located in front of the right eye of the user who wears the imagedisplay section 20 and allows the right eye to visually recognize animage. The left light guide plate 28 is located in front of the left eyeof the user who wears the image display section 20 and allows the lefteye to visually recognize an image.

The front frame 27 has a shape that links one end of the right lightguide plate 26 and one end of the left light guide plate 28 to eachother, and the linkage position corresponds to a position between theeyes of the user who wears the image display section 20. A nose pad thatcomes into contact with the nose of the user who wears the image displaysection 20 may be provided as part of the front frame 27 and in theposition where the right light guide plate 26 and the left light guideplate 28 are linked to each other. In this case, the nose pad, the rightholder 21, and the left holder 23 allow the image display section 20 tobe held on the user's head. Further, a belt (not shown) that comes intocontact with the back of the head of the user who wears the imagedisplay section 20 may be linked to the right holder 21 and the leftholder 23, and in this case, the belt allows the image display section20 to be securely held on the user's head.

The right display unit 22 displays an image via the right light guideplate 26. The right display unit 22 is provided as part of the rightholder 21 and located in the vicinity of the right temporal region ofthe user who wears the image display section 20. The left display unit24 displays an image via the left light guide plate 28. The left displayunit 24 is provided as part of the left holder 23 and located in thevicinity of the left temporal region of the user who wears the imagedisplay section 20.

The right light guide plate 26 and the left light guide plate 28 in thepresent embodiment are each an optical section, a prism, for example,made, for example, of a light transmissive resin and guide image lightoutputted from the right display unit 22 and the left display unit 24 tothe user's eyes.

A light control plate (not shown) may be provided on a surface of eachof the right light guide plate 26 and the left light guide plate 28. Thelight control plate is a thin-plate-shaped optical element havingtransmittance that varies in accordance with the range of the wavelengthof light passing therethrough and therefore functions as what is calleda wavelength filter. The light control plates are so disposed as tocover, for example, part of the front side of the front frame 27, whichis the side opposite the side facing the user's eyes. Appropriateselection of optical characteristics of the light control plates allowsadjustment of the transmittance of light in an arbitrary wavelengthrange, such as visible light, infrared light, and ultraviolet light andtherefore allows adjustment of the amount of outside light externallyincident on the right light guide plate 26 and the left light guideplate 28 and passing through the right light guide plate 26 and the leftlight guide plate 28.

The image display section 20 guides the image light produced by theright display unit 22 and the left display unit 24 to the right lightguide plate 26 and the left light guide plate 28. The image light guidedto the right light guide plate 26 and the left light guide plate 28 isincident on the right eye and left eye of the user and allows the userto visually recognize virtual images. The image display section 20 thusdisplays an image.

In a case where outside light passes through the right light guide plate26 and the left light guide plate 28 from the side in front of the userand impinges on the user's eyes, the image light that forms virtualimages and the outside light are incident on the user's eyes, and thevisibility of the virtual images is therefore affected by the intensityof the outside light. Therefore, for example, attaching the lightcontrol plates to the front frame 27 and selecting or adjusting theoptical characteristics of the light control plates as appropriate allowadjustment of the visibility of the virtual images. In a typicalexample, light control plates having light transmittance high enough toat least allow the user who wears the HMD 100 to visually recognize anoutside scene can be used. Further, when the light control plates areused, it can be expected to achieve an effect of protecting the rightlight guide plate 26 and the left light guide plate 28 and suppressingdamage of the right light guide plate 26 and the left light guide plate28, adhesion of dirt thereto, and other undesirable effects thereon. Thelight control plates may be attachable to and detachable from the frontframe 27 or the right light guide plate 26 and the left light guideplate 28. A plurality of types of light control plates may be changedfrom one to another in an attachable/detachable manner, or the lightcontrol plates may be omitted.

A camera 61 is disposed in the front frame 27 of the image displaysection 20. The configuration and arrangement of the camera 61 are sodetermined that the camera 61 performs imaging in the direction towardan outside scene visually recognized by the user who wears the imagedisplay section 20. For example, the camera 61 is provided in the frontsurface of the front frame 27 and in a position where the camera 61 doesnot block the outside light passing through the right light guide plate26 and the left light guide plate 28. In the example shown in FIG. 1,the camera 61 is disposed on the side facing the end ER of the frontframe 27. The camera 61 may instead be disposed on the side facing theend EL of the front frame 27 or in the portion where the right lightguide plate 26 and the left light guide plate 28 are linked to eachother.

The camera 61 is a digital camera including an imaging element, such asa CCD or a CMOS element, an imaging lens, and other components. Thecamera 61 in the present embodiment is a monocular camera but mayinstead be a stereocamera. The camera 61 captures an image of at leastpart of an outside scene (real space) in the direction extending fromthe front side of the HMD 100, in other words, in the direction towardthe visual field of the user who wears the HMD 100. In anotherexpression, the camera 61 performs imaging over the range or in thedirection that overlaps with the user's visual field and performsimaging in the direction in which the user gazes. The direction and sizeof the angle of view of the camera 61 can be set as appropriate. In thepresent embodiment, the angle of view of the camera 61 contains theoutside visually recognized by the user through the right light guideplate 26 and the left light guide plate 28, as will be described later.More preferably, the angle of view of the camera 61 is so set that thecamera 61 can capture an image of the user's entire visual field thatthe user can visually recognize through the right light guide plate 26and the left light guide plate 28.

The camera 61 performs imaging under the control of an imaging controlsection 145, which is provided in a control section 150 (FIG. 5). Thecamera 61 outputs captured image data to the control section 150 via aninterface 211, which will be described later.

Sightline sensors (sightline detection sections) 68 are disposed on theuser's side of the image display section 20. The sightline sensors 68are a pair of sensors provided in positions at the middle between theright light guide plate 26 and the left light guide plate 28 incorrespondence with the user's right eye RE and left eye LE. Thesightline sensors 68 are formed, for example, of a pair of cameras thatcapture images of the user's right eye RE and left eye LE. The sightlinesensors 68 perform imaging under the control of the control section 150(FIG. 5), and the control section 150 detects, from the captured imagedata, light reflected off the eyeball surfaces of the right eye RE andthe left eye LE and images of the pupils to identify the sightlinedirections.

FIG. 2 is a key part plan view showing the configuration of an opticalsystem provided in the image display section 20. FIG. 2 shows the user'sright eye RE and left eye LE for ease of description.

The right display unit 22 and the left display unit 24 have abilaterally symmetric structure, as shown in FIG. 2. As a configurationthat allows the user's right eye RE to visually recognize an image, theright display unit 22 includes an OLED (organic light emitting diode)unit 221, which emits image light, and a right optical system 251, whichincludes lens groups and other components that guide the image light Lemitted from the OLED unit 221. The image light L is guided by the rightoptical system 251 to the right light guide plate 26.

The OLED unit 221 includes an OLED panel 223 and an OLED drive circuit225, which drives the OLED panel 223. The OLED panel 223 is aself-luminous display panel formed of light emitting elements that arearranged in a matrix and emit R (red) G (green), and B (blue) colorlight fluxes on the basis of organic electroluminescence. The OLED panel223 has a plurality of pixels with each of the pixels being a lightemitting unit formed of one R element, one G element, and one B element,and the pixels arranged in a matrix form an image. The OLED drivecircuit 225 selects a light emitting element provided in the OLED panel223 and supplies the light emitting element with electric power to causethe light emitting element in the OLED panel 223 to emit light under thecontrol of the control section 150 (FIG. 5). The OLED drive circuit 225is fixed to the rear surface of the OLED panel 223, that is, the rearside of the light emitting surface, for example, in a bonding process.The OLED drive circuit 225 may be formed, for example, of asemiconductor device that drives the OLED panel 223 and mounted on asubstrate (not shown) fixed to the rear surface of the OLED panel 223. Atemperature sensor 217 is mounted on the substrate.

The OLED panel 223 may instead have a configuration in which lightemitting elements that emit white light are arranged in a matrix andcolor filters corresponding to the R, G, and B three colors are sodisposed as to be superimposed on the light emitting elements. Stillinstead, an OLED panel 223 having a WRGB configuration including lightemitting elements that radiate W (white) light in addition to the lightemitting elements that radiate the R, G, and B color light fluxes may beused.

The right optical system 251 includes a collimator lens that convertsthe image light L outputted from the OLED panel 223 into a parallelizedlight flux. The image light L having been converted by the collimatorlens into a parallelized light flux is incident on the right light guideplate 26. A plurality of reflection surfaces that reflect the imagelight L are formed along the optical path along which the light isguided in the right light guide plate 26. The image light L undergoesreflection multiple times in the right light guide plate 26 and isguided toward the right eye RE. A half-silvered mirror 261 (reflectionsurface) located in front of the right eye RE is formed on the rightlight guide plate 26. The image light L is reflected off thehalf-silvered mirror 261, exits out of the right light guide plate 26toward the right eye RE, and forms an image on the retina of the righteye RE, whereby an image is visually recognized by the user.

As a configuration that allows the user's left eye LE to visuallyrecognize an image, the left display unit 24 includes an OLED unit 241,which emits image light, and a left optical system 252, which includeslens groups and other components that guide the image light L emittedfrom the OLED unit 241. The image light L is guided by the left opticalsystem 252 to the left light guide plate 28.

The OLED unit 241 includes an OLED panel 243 and an OLED drive circuit245, which drives the OLED panel 243. The OLED panel 243 is aself-luminous display panel configured in the same manner as the OLEDpanel 223. The OLED drive circuit 245 selects a light emitting elementprovided in the OLED panel 243 and supplies the light emitting elementwith electric power to cause the light emitting element in the OLEDpanel 243 to emit light under the control of the control section 150(FIG. 5). The OLED drive circuit 245 is fixed to the rear surface of theOLED panel 243, that is, the rear side of the light emitting surface,for example, in a bonding process. The OLED drive circuit 245 may beformed, for example, of a semiconductor device that drives the OLEDpanel 243 and mounted on a substrate (not shown) fixed to the rearsurface of the OLED panel 243. A temperature sensor 239 is mounted onthe substrate.

The right optical system 252 includes a collimator lens that convertsthe image light L outputted from the OLED panel 243 into a parallelizedlight flux. The image light L having been converted by the collimatorlens into a parallelized light flux is incident on the left light guideplate 28. The left light guide plate 28 is an optical element in which aplurality of reflection surfaces that reflect the image light L areformed and is, for example, a prism. The image light L undergoesreflection multiple times in the left light guide plate 28 and is guidedtoward the left eye LE. A half-silvered mirror 281 (reflection surface)located in front of the left eye LE is formed on the left light guideplate 28. The image light L is reflected off the half-silvered mirror281, exits out of the left light guide plate 28 toward the left eye LE,and forms an image on the retina of the left eye LE, whereby an image isvisually recognized by the user.

The HMD 100 having the configuration described above functions as asee-through-type display apparatus. That is, on the user's right eye REare incident the image light L having been reflected off thehalf-silvered mirror 261 and outside light OL having passed through theright light guide plate 26. On the user's left eye LE are incident theimage light L having been reflected off the half-silvered mirror 281 andoutside light OL having passed through the half-silvered mirror 281. TheHMD 100 thus causes the image light L carrying internally processedimages and the outside light OL to be superimposed on each other andcauses the superimposed light to be incident on the user's eyes, and theuser views an outside scene through the right light guide plate 26 andthe left light guide plate 28 and visually recognizes images formed bythe image light L and superimposed on the outside scene.

The half-silvered mirrors 261 and 281 are image extracting sections thatreflect the image light outputted from the right display unit 22 and theleft display unit 24 and extract images from the image light, and it canbe said that the half-silvered mirrors 261 and 281 form the displaysection.

The left optical system 252 and the left light guide plate 28 are alsocollectively called a “left light guide unit,” and the right opticalsystem 251 and the right light guide plate 26 are also collectivelycalled a “right light guide unit.” The configuration of the right andleft light guide units is not limited to the example described above andcan be arbitrarily configured as long as the image light is used to formvirtual images in front of the user's eyes. A diffraction grating or ahalf-transmissive/reflective film may, for example, be used as each ofthe right and left light guide units.

Referring back to FIG. 1, the control apparatus 10 and the image displaysection 20 are connected to each other via a connection cable 40. Theconnection cable 40, which is detachably connected to a connector (notshown) provided on the lower side of the case 10A, is inserted throughthe end of the left holder 23 and connected to a variety of circuitsprovided in the image display section 20. The connection cable 40includes a metal cable or an optical fiber cable through which digitaldata is transmitted and may further include a metal cable through whichan analog signal is transmitted. A connector 46 is provided in a halfwayposition along the connection cable 40. The connector 46 is a jack towhich a stereo mini plug is connected, and the connector 46 and thecontrol apparatus 10 are connected to, for example, via a line throughwhich an analog voice signal is transmitted. In the configurationexample shown in FIG. 1, a headset 30 including a right earphone 32 anda left earphone 34, which form a stereo headphone, and a microphone 63is connected to the connector 46.

The control apparatus 10 and the image display section 20 may bewirelessly connected to each other. For example, the control apparatus10 and the image display section 20 may be configured to transmit andreceive control signals and data to and from each other over wirelesscommunication compliant, for example, with Bluetooth (registeredtrademark), a wireless LAN (including WiFi (registered trademark)), orany other standard.

The microphone 63 is so disposed that a sound collector of themicrophone 63 faces along the user's sightlines as shown, for example,in FIG. 1 and collects voice and outputs a voice signal to a voiceinterface 182 (FIG. 4). The microphone 63 may, for example, be amonaural microphone, a stereo microphone, a directional microphone, oran omni-directional microphone.

The control apparatus 10 includes, as the operation sections operated bythe user, the buttons 11, the LED indicator 12, the trackpad 14, theup/down keys 15, the changeover switch 16, and the power switch 18. Theoperation sections are disposed on the surface of the case 10A.

The buttons 11 are formed of keys and switches for operating the controlapparatus 10, and the keys and switches are displaced when they arepressed. For example, the buttons 11 are formed of a menu key, a homekey, and a “return” key for operation relating to an operation system141 (FIG. 5) and other types of software executed by the controlapparatus 10.

The LED indicator 12 illuminates or blinks in correspondence with theaction state of the HMD 100. The up/down keys 15 are used to input aninstruction of increase and decrease of the magnitude of sound outputtedfrom the right earphone 32 and the left earphone 34 and input aninstruction of increase and decrease of the brightness of an imagedisplayed by the image display section 20. The changeover switch 16 is aswitch that switches an input corresponding to operation of one of theup/down keys 15 to an input corresponding to operation of the other oneof the up/down keys 15 and vice versa. The power switch 18 is a switchthat switches the power-on state and the power-off state of the HMD 100from one to the other and is, for example, of a slide switch.

The trackpad 14 has an operation surface that detects contact operationand outputs an operation signal in accordance with operation performedon the operation surface. A method for detecting operation performed onthe operation surface is not limited to a specific method and can be anelectrostatic method, a pressure detection method, an optical method, orany other method. Contact operation performed on the trackpad 14 (touchoperation) is detected with a touch sensor (not shown). The trackpad 14is provided with an LED display section 17. The LED display section 17includes a plurality of LEDs, and light from each of the LEDs passesthrough the trackpad 14 and displays icons or other symbols foroperation. The icons or other symbols each function as a softwarebutton.

FIG. 3 is a perspective view showing the configuration of the imagedisplay section 20 and shows a key part configuration when the imagedisplay section 20 is viewed from the side facing the user's head. FIG.3 shows one side of the image display section 20, that is, the sidefacing the user's head, in other words, the side visible to the user'sright eye RE and left eye LE. In another expression, FIG. 3 shows therear side of the right light guide plate 26 and the left light guideplate 28.

In FIG. 3, the half-silvered mirror 261, via which the user's right eyeRE is irradiated with image light, and the half-silvered mirror 281, viawhich the user's left eye LE is irradiated with image light, are eachviewed as a roughly quadrangular region. The entire right light guideplate 26 and left light guide plate 28 including the half-silveredmirrors 261 and 281 transmit outside light, as described above. The usertherefore visually recognizes an outside scene through the entire rightlight guide plate 26 and left light guide plate 28 and further visuallyrecognizes rectangular displayed images in the positions of thehalf-silvered mirrors 261 and 281.

The camera 61 is disposed in a right end portion of the image displaysection 20 and performs imaging in the direction in which the user's twoeyes are oriented, that is, captures an image of a space in front of theuser. The optical axis of the camera 61 falls within the rangecontaining the directions of the sightlines extending from the right eyeRE and the left eye LE. The outside scene visually recognizable by theuser who wears the HMD 100 is not limited to the infinity. For example,when the user gazes at a target object located in front of the user withthe two eyes, the distance from the user to the target object rangesfrom about 30 cm to 10 m in many cases, and the distance is more likelyto range from about 1 to 4 m. In view of the fact described above,guideline values of the upper and lower limits of the distance from theuser to the target object in typical conditions under which the HMD 100is used may be set. The guideline values may be determined byinvestigation or experiment or may be set by the user. The optical axisand the angle of view of the camera 61 are preferably so set that thetarget object falls within the angle of view in a case where thedistance to the target object under the typical use condition is equalto the set guideline value of the upper limit and in a case where thedistance is equal to the set guideline value of the lower limit.

In general, it is believed that a person's angular field of view isabout 200 degrees in the horizontal direction and about 125 degrees inthe vertical direction. Within these ranges, an effective field of view,where the person has excellent information reception capability, extendsover a horizontal range of about 30 degrees and a vertical range ofabout 20 degrees. It is further believed that a stable field offixation, where a point of fixation at which the person gazes is viewedin a quick, stable manner, extends over a horizontal range from about 60to 90 degrees and a vertical range from about 45 to 70 degrees. In acase where the point of fixation coincides with the target objectlocated in front of the user, the effective field of view within theuser's field of view extends over the horizontal range of about 30degrees and the vertical range of about 20 degrees around the sightlinesextending from the right eye RE and the left eye LE. The stable field offixation extends over the horizontal range from about 60 to 90 degreesand the vertical range from about 45 to 70 degrees around thesightlines, and the angular field of view extends over the horizontalrange from about 200 degrees and the vertical range from about 125degrees around the sightlines. The actual field of view visuallyrecognized by the user through the right light guide plate 26 and theleft light guide plate 28 can be called an actual field of view (FOV).The actual field of view corresponds to the field of view actuallyvisually recognized by the user through the right light guide plate 26and the left light guide plate 28 in the configuration of the presentembodiment shown in FIGS. 1 and 2. The actual field of view is narrowerthan the angular field of view and the stable field of fixation butwider than the effective field of view.

The angle of view of the camera 61 preferably allows imaging over arange wider than the user's field of view. Specifically, the angle ofview is preferably wider than at least the user's effective field ofview. The angle of view is more preferably wider than the user's actualfield of view. The angle of view is further preferably wider than theuser's stable field of fixation and is most preferably wider than theuser's binocular angular field of view.

The camera 61 may include what is called a wide-angle lens as theimaging lens for imaging over a wide angle of view. The wide-angle lensmay include a lens called a super-wide-angle lens or a semi-wide-anglelens. The camera 61 may include a fixed-focal-length lens, a zoom lens,or a lens group formed of a plurality of lenses.

FIG. 4 is a block diagram showing the configuration of each portion thatforms the HMD 100.

The control apparatus 10 includes a main processor 140, which executes aprogram to control the HMD 100. A memory 118 and a nonvolatile storagesection 121 are connected to the main processor 140. Further, thetrackpad 14 and an operation section 110 are connected as input devicesto the main processor 140. A six-axis sensor 111 and a magnetic sensor113 are connected as sensors to the main processor 140. A GPS (globalpositioning system) receiver 115, a communication section 117, a voicecodec 180, an external connector 184, an external memory interface 186,a USB connector 188, a sensor hub 192, and an FPGA 194 are connected tothe main processor 140. The portions described above function asinterfaces to an external apparatus.

The main processor 140 is mounted on a controller substrate 120 built inthe control apparatus 10. The memory 118, the nonvolatile storagesection 121, and other components may be mounted on the controllersubstrate 120 in addition to the main processor 140. In the presentembodiment, the six-axis sensor 111, the magnetic sensor 113, the GPSreceiver 115, the communication section 117, the memory 118, thenonvolatile storage section 121, the voice codec 190, and othercomponents are mounted on the controller substrate 120. The externalconnector 184, the external memory interface 186, the USB connector 188,the sensor hub 192, the FPGA 194, and an interface 196 may further bemounted on the controller substrate 120.

The memory 118 forms a work area that is used when the main processor140 executes a program, and the memory 118 temporarily stores theprogram executed by the main processor 140 and data processed by themain processor 140. The nonvolatile storage section 121 is formed of aflash memory or an eMMC (embedded multimedia card). The nonvolatilestorage section 121 stores the program executed by the main processor140 and a variety of data processed when the main processor 140 executesthe program.

The main processor 140 detects contact operation performed on theoperation surface of the trackpad 14 on the basis of the operationsignal inputted from the trackpad 14 and acquires an operation position.

The operation section 110 includes the buttons 11 and the LED displaysection 17. When any of operation components, such as the buttons thatform the buttons 11 and switches, is operated, the operation section 110outputs an operation signal corresponding to the operated operationcomponent to the main processor 140.

The LED display section 17 controls the LED indicator 12 to start andstop illuminating under the control of the main processor 140. The LEDdisplay section 17 may include LEDs (not shown) disposed immediatelybelow the trackpad 14 and a drive circuit that turns on the LEDs. Inthis case, the LED display section 17 causes the LEDs to startilluminating, blink, and stop illuminating under the control of the mainprocessor 140.

The six-axis sensor 111 is a motion sensor (inertia sensor) including athree-axis acceleration sensor and a three-axis gyro (angular velocity)sensor. The six-axis sensor 111 may be an IMU (inertial measurementunit) that incorporates the sensors described above in a modular form.

The magnetic sensor 113 is, for example, a three-axis geomagneticsensor.

The six-axis sensor 111 and magnetic sensor 113 output detection valuesto the main processor 140 in accordance with sampling frequenciesspecified in advance. The six-axis sensor 111 and magnetic sensor 113output the detection values to the main processor 140 in response to arequest from the main processor 140 at the timing specified by the mainprocessor 140.

The GPS receiver 115 includes a GPS antenna that is not shown andreceives GPS signals transmitted from GPS satellites. The GPS receiver115 outputs the received GPS signals to the main processor 140. The GPSreceiver 115 further measures signal strengths of the received GPSsignals and outputs the measured signal strengths to the main processor140. The signal strengths can, for example, be expressed in the form ofinformation on the received signal strength indication (RSSI), theelectric field strength, or the magnetic field strength, thesignal-to-noise ratio (SNR).

The communication section 117 performs wireless communication with anexternal apparatus. The communication section 117 includes an antenna,an RF circuit, a baseband circuit, a communication control circuit, andother components or is formed of a device in which the componentsdescribed above are integrated with one another. The communicationsection 117 performs wireless communication compliant, for example, withBluetooth, a wireless LAN standard (including Wi-Fi), or any otherstandard.

The voice interface 182 is an interface that inputs and outputs a voicesignal. In the present embodiment, the voice interface 182 includes theconnector 46 (FIG. 1), with which the connection cable 40 is provided.The connector 46 is connected to the headset 30. The voice signaloutputted via the voice interface 182 is inputted to the right earphone32 and the left earphone 34, and the right earphone 32 and the leftearphone 34 then output voice. The microphone 63, with which the headset30 is provided, collects voice and outputs a voice signal to the voiceinterface 182. The voice signal inputted from the microphone 63 to thevoice interface 182 is inputted to the external connector 184.

The voice codec 180 is connected to the voice interface 182 anddecodes/encodes a voice signal inputted and outputted via the voiceinterface 182. The voice codec 180 may include an A/D converter thatconverts an analog voice signal into digital voice data and a D/Aconverter that performs conversion in the opposite direction. Forexample, the HMD 100 in the present embodiment outputs voice via theright earphone 32 and the left earphone 34 and collects voice via themicrophone 63. The voice codec 180 converts digital voice data outputtedby the main processor 140 into an analog voice signal and outputs thesignal via the voice interface 182. The voice codec 180 further convertsan analog voice signal inputted to the voice interface 182 into digitalvoice data and outputs the data to the main processor 140.

The external connector 184 is a connector that connects the mainprocessor 140 to an external apparatus that communicates with the mainprocessor 140. The external connector 184 is an interface that connectsthe external apparatus to the main processor 140, for example, when thecomputer program executed by the main processor 140 is debugged oraction logs of the HMD 100 are collected.

The external memory interface 186 is an interface that allows connectionto a portable memory device and includes, for example, a memory cardslot, into which a card-shaped recording medium is inserted and viawhich data can be read, and an interface circuit. The size, shape, andstandard of the card-shaped recording medium in this case are notlimited to specific ones and can be changed as appropriate.

The USB (universal serial bus) connector 188 includes a connectorcompliant with the USB standard and an interface circuit. The USBconnector 188 allows connection to a USB memory device, a smartphone, acomputer, and other devices. The size, shape, and compatible USBstandard version of the USB connector 188 can be selected and changed asappropriate.

The sensor hub 192 and the FPGA 194 are connected to the image displaysection 20 via the interface (I/F) 196. The sensor hub 192 acquiresdetection values from the variety of sensors provided in the imagedisplay section 20 and outputs the detection values to the mainprocessor 140. The FPGA 194 processes data transmitted from the mainprocessor 140 and received by the portions of the image display section20 and vice versa and transports the processed data via the interface196.

The HMD 100 further includes a vibrator 19. The vibrator 19 includes amotor and an off-center rotator (neither of them is shown) and mayinclude other necessary configurations. Rotary operation of the motordescribed above under the control of the main processor 140 allows thevibrator 19 to produce vibration. The HMD 100 causes the vibrator 19 toproduce vibration in a predetermined vibration pattern when operationperformed on the operation section 110 is detected, when the HMD 100 ispowered on and off, or in other cases.

The right display unit 22 and the left display unit 24 of the imagedisplay section 20 are each connected to the control apparatus 10. Inthe HMD 100, the connection cable 40 is connected to the left holder 23,as shown in FIG. 1, and a wiring line connected to the connection cable40 is routed in the image display section 20, whereby the right displayunit 22 and the left display unit 24 are each connected to the controlapparatus 10.

The right display unit 22 includes a display unit substrate 210. On thedisplay unit substrate 210 are mounted an interface (I/F) 211, which isconnected to the interface 196, a receiver (Rx) 213, which receives datainputted from the control apparatus 10 via the interface 211, and anEEPROM 215.

The interface 211 connects the receiver 213, the EEPROM 215, thetemperature sensor 217, the camera 61, an illuminance sensor 65, and anLED indicator 67 to the control apparatus 10.

The EEPROM (electrically erasable read-only memory) 215 stores a varietyof types of data in the form readable by the main processor 140. TheEEPROM 215 stores, for example, data on light emission characteristicsand display characteristics of the OLED units 221 and 241 provided inthe image display section 20, data on the characteristics of the sensorprovided in the right display unit 22 or the left display unit 24, andother data. Specifically, the EEPROM 215 stores a parameter involved ingamma correction of the OLED units 221 and 241, data used to compensatedetection values from the temperature sensors 217 and 239, and otherdata. These data are produced when the HMD 100 is inspected just beforethe HMD 100 is shipped from the factory and written onto the EEPROM 215.After the shipment, the main processor 140 uses the data in the EEPROM215 to carry out the processes described above.

The camera 61 performs imaging in accordance with a signal inputted viathe interface 211 and outputs captured image data or a signalrepresenting an imaging result to the control apparatus 10.

The illuminance sensor 65 is provided at the end ER of the front frame27 and so disposed as to receive outside light from a space in front ofthe user who wears the image display section 20, as shown in FIG. 1. Theilluminance sensor 65 outputs a detection value corresponding to theamount of light received by the illuminance sensor 65 (intensity oflight received by illuminance sensor 65).

The LED indicator 67 is disposed at the end ER of the front frame 27 andin the vicinity of the camera 61, as shown in FIG. 1. The LED indicator67 illuminates when the camera 61 is performing imaging to notify theuser that the imaging is underway. The camera 61 may be a monocularcamera or a stereocamera.

The temperature sensor 217 performs temperature detection and outputs,as a detection value, a voltage value or a resistance valuecorresponding to the detected temperature. The temperature sensor 217 ismounted on the rear side of the OLED panel 223 (FIG. 2). The temperaturesensor 217 may instead be mounted, for example, on the substrate onwhich the OLED drive circuit 225 is mounted. In the configurationdescribed above, the temperature sensor 217 primarily detects thetemperature of the LED panel 223.

The receiver 213 receives data transmitted by the main processor 140 viathe interface 211. The receiver 213, when it receives image data on animage to be displayed by the OLED unit 221, outputs the received imagedata to the OLED drive circuit 225 (FIG. 2).

The left display unit 24 includes a display unit substrate 210. On thedisplay unit substrate 210 are mounted an interface (I/F) 231, which isconnected to the interface 196, and a receiver (Rx) 233, which receivesdata inputted from the control apparatus 10 via the interface 231. Asix-axis sensor 235 and a magnetic sensor 237 are mounted on the displayunit substrate 210. The magnetic sensor 237 may instead be disposed onthe side facing the outer surface of the enclosure of the image displaysection 20. In a case where the enclosure of the image display section20 is made of metal, noise is superimposed on a detection value from themagnetic sensor 237 and the detection accuracy of the magnetic sensor237 lowers in some cases. The magnetic sensor 237 can therefore bedisposed on the side facing the outer surface of the enclosure of theimage display section 20, whereby decrease in the detection accuracy ofthe magnetic sensor 237 can be avoided.

The interface 231 connects the receiver 233, the six-axis sensor 235,the magnetic sensor 237, and the temperature sensor 239 to the controlapparatus 10.

The six-axis sensor 235 is a motion sensor (inertia sensor) including athree-axis acceleration sensor and a three-axis gyro (angular velocity)sensor. The six-axis sensor 235 may be an IMU that incorporates thesensors described above in a modular form.

The magnetic sensor 237 is, for example, a three-axis geomagneticsensor.

The temperature sensor 239 performs temperature detection and outputs,as a detection value, a voltage value or a resistance valuecorresponding to the detected temperature. The temperature sensor 239 ismounted on the rear side of the OLED panel 243 (FIG. 2). The temperaturesensor 239 may instead be mounted, for example, on the substrate onwhich the OLED drive circuit 245 is mounted. In the configurationdescribed above, the temperature sensor 239 primarily detects thetemperature of the OLED panel 243.

The temperature sensor 239 may instead be built in the OLED panel 243 orthe OLED drive circuit 245. The substrate described above may be asemiconductor substrate. Specifically, in a case where the OLED panel243 is implemented as an Si-OLED, along with the OLED drive circuit 245and other components, in the form of an integrated circuit on a unifiedsemiconductor chip, the temperature sensor 239 may be implemented in thesemiconductor chip.

The camera 61, the illuminance sensor 65, and the temperature sensor 217provided in the right display unit 22, and the six-axis sensor 235, themagnetic sensor 237, and the temperature sensor 239 provided in the leftdisplay unit 24 are connected to the sensor hub 192. The sensor hub 192sets and initializes, under the control of the main processor 140, thesampling cycle in accordance with which each of the sensors performsdetection. The sensor hub 192, for example, conducts electricity to eachof the sensors, transmits control data thereto, and acquires a detectionvalue therefrom in accordance with the sampling cycle in accordance withwhich the sensor performs detection. The sensor hub 192 outputsdetection values from the sensors provided in the right display unit 22and the left display unit 24 at preset timings to the main processor140. The sensor hub 192 may have the function of temporarily holding thedetection value from each of the sensors in accordance with the timingwhen the detection value is outputted to the main processor 140. Thesensor hub 192 may have the function of converting data in differentformats into data in a unified format in correspondence with differencesin the signal format or data format of the detection values from thesensors and outputting the converted data to the main processor 140.

The sensor hub 192 starts or stops conducting electricity to the LEDindicator 67 under the control of the main processor 140 to turn on theLED indicator 67 or cause the LED indicator 67 to blink insynchronization with the timing when the camera 61 starts or stopsimaging.

The control apparatus 10 includes a power supply section 130 andoperates by using electric power supplied from the power supply section130. The power supply section 130 includes a chargeable battery 132 anda power supply control circuit 134, which detects the amount ofremaining electric power in the battery 132 and controls electricitycharge of the battery 132. The power supply control circuit 134 isconnected to the main processor 140 and outputs a detection valuerepresenting the amount of remaining electric power in the battery 132or a detection value representing the voltage across the battery 132 tothe main processor 140. Electric power may be supplied from the controlapparatus 10 to the image display section 20 on the basis of theelectric power supplied from the power supply section 130. The mainprocessor 140 may be configured to be capable of controlling the stateof electric power supply from the power supply section 130 to not onlythe portions of the control apparatus 10 but the image display section20.

FIG. 5 is a functional block diagram of a storage section 122 and thecontrol section 150, which form the control system of the controlapparatus 10. The storage section 122 shown in FIG. 5 is a logicalstorage section formed of the nonvolatile storage section 121 (FIG. 4)and may include the EEPROM 215. The control section 150 and a variety offunctional portions provided in the control section 150 are achievedwhen the main processor 140 executes the program, that is, when theprogram and the hardware cooperate with each other. The control section150 and the functional portions that form the control section 150 areformed, for example, of the main processor 140, the memory 118, and thenonvolatile storage section 121. The control section 150 corresponds tothe “processing section” in an aspect of the invention.

The control section 150 uses data stored in the storage section 122 tocarry out a variety of processes to control the HMD 100. The storagesection 122 stores a variety of data to be processed by the controlsection 150. Specifically, the storage section 122 stores setting data123, content data 124, and display position information 125.

The setting data 123 contains a variety of setting values used to setthe action of the HMD 100. In a case where the control section 150 usesa parameter, a determinant, an arithmetic expression, an LUT (lookuptable), and other factors when the control section 150 controls the HMD100, the setting data 123 may contain the parameter and the like.

The content data 124 is data displayed by the image display section 20under the control of the control section 150 and contains, for example,document files, such as those created by Word and Excel, still imagedata, and motion image data. The content data 124 may further containvoice data. The content data 124 may be 3D image data that containsparallax and allows the right display unit 22 and the left display unit24 to display images with the parallax therebetween. The content data124 may be image data for displaying images of operation buttons, a GUI(graphical user interface), a software keyboard, and other objectsdisplayed by using AR (augmented reality).

The content data 124 may be a bidirectional content that causes theimage display section 20 to display the content, causes the controlapparatus 10 to accept the user's operation, and causes the controlsection 150 to carry out a process according to the acquired operation.In this case, the content data 124 may contain image data on a menuscreen displayed to accept operation, data that specifies a process orany other action corresponding to each item contained in the menuscreen, and other data.

The display position information 125 is information representingrelative display positions of a plurality of contents. The displayposition information 125, which will be described later in detail, isformed of information for identifying an application program(hereinafter referred to as application) that displays a content andinformation representing the display size of the content displayed bythe application and the coordinates to which the content is related in aspherical coordinate system. The spherical coordinate system will bedescribed later in detail. The display position information 125 furthercontains information representing a reference orientation set as areference state, which will be described later, and the angle ofelevation or the angle of depression representing the inclination of theuser's head.

The control section 150 includes, as functional blocks, an operatingsystem (OS) 141, an image processing section 143, an imaging controllingsection 145, an operation controlling section 147, a pointing elementdetecting section 149, a head motion detecting section 151, a referencesetting section 153, a display controlling section 155, and a coordinatesetting section 157. The functional blocks represent functions achievedwhen a CPU develops the program in a RAM and executes the developedprogram and expressed in the form of blocks for convenience. The controlsection 150 can also function, in addition to the functional blocks, asa processing section that provides a plurality of other functions. Forexample, the control section 150 may be allowed to operate as an editingprocessing section that changes the substance of a content displayed bythe image display section 20.

The function of the operating system 141 is the function of a controlprogram stored in the storage section 122, and the other sections areeach the function of an application program executed on the operatingsystem 141.

The image processing section 143 produces signals to be transmitted tothe right display unit 22 and the left display unit 24 on the basis ofimage data on still images or video images to be displayed by the imagedisplay section 20. The signals produced by the image processing section143 may be a vertical sync signal, a horizontal sync signal, a clocksignal, an analog image signal, and other signals.

The image processing section 143 may carry out a resolution conversionprocess of converting the resolution of the image data into resolutionsuitable for the right display unit 22 and the left display unit 24. Theimage processing section 143 may further carry out an image adjustmentprocess of adjusting the luminance and chroma of the image data, a 2D/3Dconversion process of creating 2D image data from 3D image data orcreating 3D image data from 2D image data, and other processes. Havingcarried out the image processing described above, the image processingsection 143 produces signals for displaying images on the basis of theprocessed image data and transmits the signals to the image displaysection 20 via the connection cable 40.

The image processing section 143 is not necessarily achieved by theprogram executed by the main processor 140 and may instead be formed ofhardware (DSP (digital signal processor), for example) separate from themain processor 140.

The imaging control section 145 controls the camera 61 to cause it toperform imaging for generation of captured image data. The imagingcontrol section 145 causes the storage section 122 to temporarily storethe produced captured image data. In a case where the camera 61 isconfigured as a camera unit including a circuit that produces capturedimage data, the imaging control section 145 acquires the captured imagedata from the camera 61 and causes the storage section 122 totemporarily store the data.

The operation controlling section 147 detects operation performed on thetrackpad 14 and the operation section 110 and outputs data correspondingto the detected operation. For example, when any of the buttons or anyother component on the operation section 110 is operated, the operationcontrolling section 147 produces operation data representing the contentof the operation and outputs the operation data to the displaycontrolling section 155. The display controlling section 155 changes thestate of images displayed by the image display section 20 in accordancewith the operation data inputted from the operation controlling section147.

When operation performed on the trackpad 14 is detected, the operationcontrolling section 147 successively acquires the coordinates of theoperation position on the trackpad 14. The operation controlling section147 produces the trajectory of the operation positions. The trajectoryof the operation positions is the trajectory of positions whereoperation is performed during the period for which touch operation onthe trackpad 14 is not terminated, that is, during the period for whichthe state in which the operation element is in contact with the trackpad14 continues.

The pointing element detecting section 149 acquires captured image datafrom the storage section 122 and detects a region of the user's hand andfingers (hereinafter referred to as hand/finger) (hereinafter referredto as hand/finger region) contained in the acquired captured image data.The pointing element detecting section 149 identifies the shape andposition of the hand/finger seen in the detected hand/finger region. Forexample, the pointing element detecting section 149 extracts aflesh-color region from the captured image data region in a flesh-colorextraction process and compares the pixel values in the extractedflesh-color region with a threshold in a threshold process to detect thehand/finger region. In the present embodiment, the description will bemade of a case where the pointing element detecting section 149 detectsthe user's hand/finger as a pointing element including part of theuser's body, but the pointing element detected by the pointing elementdetecting section 149 may instead be the user's whole arm. In a casewhere the user grasps a pointing rod or any other object with a hand,the pointing element detecting section 149 may detect the pointing rodas the pointing element or may detect the user's hand/finger includingthe pointing rod.

The head motion detecting section 151 detects the position and motion ofthe user's head on the basis of detection values from the six-axissensor 235 and the magnetic sensor 237. The user's head corresponds tothe “detection target section” in an aspect of the invention. Thedetection target section may instead be the user's upper body includingthe head. The six-axis sensor 235 and the magnetic sensor 237 performsampling every predetermined cycle (50 msec, for example) and outputanalog voltage values representing results of the detection. Theoutputted voltage values are converted with an A/D converter (not shown)into digital signals and outputted to the control section 150. Thesix-axis sensor 235 and the magnetic sensor 237 correspond to the“detection section” of an aspect of the invention.

Angular velocity sensors provided in the six-axis sensor 235 detectrotation around the X axis shown in FIG. 1 (pitch angle), rotationaround the Y axis (yaw angle), and rotation around the Z axis (rollangle) at a measurement reference point of the built-in detectionmechanism. The X, Y, and Z axes are three axes perpendicular to oneanother, as shown in FIG. 1, with the Z-axis direction corresponding tothe vertical direction, the X-axis direction corresponding to therightward/leftward direction of the user's head, and the Y-axisdirection corresponding to the frontward/rearward direction of theuser's head.

The head motion detecting section 151 detects rotation around the X axis(pitch angle) on the basis of detection values from the six-axis sensor235 to detect the user's motion of raising or lowering the face. Thehead motion detecting section 151 further detects rotation around the Yaxis (yaw angle) on the basis of detection values from the six-axissensor 235 to detect the user's motion of inclining the neck toward theright or left shoulder. The head motion detecting section 151 furtherdetects rotation around the Z axis (roll angle) on the basis ofdetection values from the six-axis sensor 235 to detect the user'smotion of turning the face clockwise or counterclockwise. In thefollowing description, the motion of raising or lowering the face, themotion of inclining the neck toward the right or left shoulder, and themotion of turning the face clockwise or counterclockwise arecollectively called head motion. Further, the X, Y, and Z axescorrespond to the “axes of rotation of the detection target section” inan aspect of the invention.

The magnetic sensor 237 is a three-axis magnetic sensor that detectsmagnetic fields in the X-axis, Y-axis, and Z-axis directions describedabove. The head motion detecting section 151 detects the direction inwhich the user faces, that is, the orientation corresponding to theregion in front of the user on the basis of detection values from themagnetic sensor 237.

The reference setting section 153 sets a reference state on the basis ofat least one of the user's head position and motion detected by thesix-axis sensor 235 and the magnetic sensor 237. The reference settingsection 153 sets the orientation that serves as the reference of thehead position or motion (hereinafter referred to as referenceorientation) as the reference state. When rotation or any other motionof the user's head is detected, the control section 150 changes thedisplay state of a content displayed by the image display section 20with reference to the set reference orientation.

To cause the HMD 100 to set the reference orientation, the user performsfirst operation and second operation. The first operation is operationthat causes the HMD 100 to set the reference state. For example, theuser performs the first operation by performing predetermined operationon the operation surface of the trackpad 14. The first operation is notlimited to operation performed on the operation surface of the trackpad14 and may, for example, be operation performed on any of the buttons 11or the up/down keys 15 or operation of tapping the operation surface ofthe trackpad 14. The first operation may still instead be the user'saction of saying “set reference orientation.”

Having performed the first operation, the user directs the head and thebody in a direction that the user desires to set as the referenceorientation. At this point, the user's head may face obliquely upward ordownward. Having faced in the direction that the user desires to set asthe reference orientation, the user performs the second operation. Thesecond operation is preferably operation that causes smallest possibleuser's head motion so that no shift in the set reference orientationoccurs. The second operation may, for example, be preset operationperformed on the trackpad 14 or the operation section 110.

The reference setting section 153, when it determines on the basis ofdetection values from the six-axis sensor 235 that the user's head hasbeen stationary for at least a predetermined period, may set theorientation in which the stationary head faces as the referenceorientation. That is, the reference orientation may be set with nosecond operation performed.

The second operation may instead be operation performed on an operationdevice 3 worn on the user.

The operation device 3 will now be described. FIG. 6 is a block diagramshowing the configuration of the operation device 3.

The operation device 3 will now be described.

The operation device 3 includes a control section 310, which controlseach portion of the operation device 3. The control section 310 includesa CPU, a ROM, a RAM, and other components that are not shown andexecutes a program stored in the ROM to achieve the function of theoperation device 3.

The operation device 3 includes a wireless communication section 321 andan NFC (near field communication) section 322 as functional portions forcommunication.

The wireless communication section 321 performs wireless datacommunication compliant with a wireless LAN (WiFi (registeredtrademark)), Miracast (registered trademark), Bluetooth, or any otherstandard. The NFC section 322 performs short-range wirelesscommunication compliant with the NFC standard.

An LCD (liquid crystal display) 322, a touch panel 324, a camera 325,and an LED (light emitting diode) 325 are connected to the controlsection 310.

The control section 310 controls the LCD 323 to cause it to display animage and letters. The touch panel 324 is so disposed as to be overlaidon the surface of the LCD 323 and detects contact operation performed onthe surface of the LCD 323. The touch panel 324 can, for example, be anelectrostatic capacitance or pressure sensitive sensor. The controlsection 310 detects contact operation via the touch panel 324 andidentifies an operation position.

The control section 310 further controls the camera 325 to cause it toperform imaging to produce captured image data. The control section 310further causes current to flow through the LED 326 and stops supplyingthe LED 326 with the current so that the LED 326 illuminates or blinksat an arbitrary timing.

A microphone 327, a button operation section 328, and a loudspeaker 329are connected to the control section 310. The microphone 327 may be amonaural microphone, a stereo microphone, a directional microphone, oran omni-directional microphone. The control section 310 acquires ananalog voice signal representing voice collected with the microphone 327to produce digital voice data.

The button operation section 328 detects operation performed on any ofbuttons provided on the operation device 3 and outputs an operationsignal corresponding to the operated button to the control section 310.

The loudspeaker 329 outputs voice on the basis of a voice signaloutputted by the control section 310.

The description of the first operation and the second operationdescribed above will be resumed. Having faced in the direction that theuser desires to set as the reference orientation, the user operates theoperation device 3. The operation of the operation device 3 may, forexample, be operation performed on a switch or a button provided on theoperation device 3 or operation of knocking the surface of the operationdevice 3. The operation device 3, when it detects a knock sound via themicrophone 327, transmits a signal representing that the HMD 100 hasaccepted the second operation.

The reference setting section 153, when it accepts the second operation,acquires detection values from the six-axis sensor 235 and the magneticsensor 237. The reference setting section 153 detects the axis ofrotation of the head and the position of the head that rotates aroundthe axis of rotation on the basis of the acquired detection values toset the reference orientation. For example, the reference settingsection 153 detects the orientation in which the user's head faces(orientation angle) on the basis of detection values from the magneticsensor 237. In this case, the axis of rotation of the head is the Z axis(vertical direction), and the position of the head that rotates aroundthe axis of rotation is the orientation angle detected on the basis ofthe detection values from the magnetic sensor 237.

The reference setting section 153 may instead detect the angle ofelevation or depression, which is the inclination of the head, as thereference state. The reference setting section 153 detects the angle ofrotation of the head around the X axis (pitch angle) on the basis ofdetection values from the angular velocity sensors provided in thesix-axis sensor 235. The detected angle of rotation around the X axis isthe inclination (angle of elevation or depression) with respect to theplane containing the X axis set along the rightward/leftward directionof the head and the Y axis set along the frontward/rearward direction ofthe head (hereinafter referred to as horizontal plane). In this case,the axis of rotation of the head is the X axis, and the position of thehead is the detected angle of elevation or depression.

The reference setting section 153 sets the detected orientation angle asthe reference orientation and causes the storage section 122 to storeinformation representing the axis of rotation of the head and thereference orientation and information representing the angle ofelevation or depression as the display position information 125.

The reference setting section 153 may set the reference state on thebasis of the user's sightline directions detected with the sightlinesensors 68 as well as detection values from the six-axis sensor 235 andthe magnetic sensor 237. Setting the reference state on the basis of theuser's sightline directions allows the user's sightline directions to beset as the reference state and therefore allows a content to be sodisplayed as to be located in the user's sightline directions.

Further, as the detection section that detects the position or motion ofthe head, the GPS receiver 115 or the camera 61 may be used in additionto the six-axis sensor 235 and the magnetic sensor 237. Using thesefunctional sections as the detection section allows an increase in theaccuracy in detection of the position and motion of the head.

The reference setting section 153 may set, as the reference direction,the direction in which the strength of the received electromagnetic waveused in the short-range wireless communication, such as Bluetooth andBLE (Bluetooth Low Energy), and transmitted from a transmitter (notshown) is maximized. Still instead, the reference state may be set onthe basis of operation performed on the operation device 3 in place ofthe action of the user's head. For example, the user directs the surfaceof the operation device 3 in a direction set as the referenceorientation. The operation device 3, in which a six-axis sensor and amagnetic sensor are incorporated, transmits detection values from thesensors to the HMD 100. The reference setting section 153 receives thedetection values detected with the sensors and transmitted from theoperation device 3 via the communication section 117 and sets thereference state on the basis of the received detection values.

The display controlling section 155 produces control signals thatcontrol the right display unit 22 and the left display unit 24, and thecontrol signals control the right display unit 22 and the left displayunit 24 to cause them to produce and output image light. Specifically,the display controlling section 155 controls the OLED drive circuits 225and 245 to cause the OLED panels 223 and 243 to display images. Thedisplay controlling section 155 controls the timing when the OLED drivecircuits 225 and 245 draw images in the OLED panels 223 and 243,controls the luminance of the images displayed by the OLED panels 223and 243, and performs other types of control on the basis of the signalsoutputted by the image processing section 143.

The display controlling section 155 further performs calibration toproduce a parameter that relates the coordinates in a display region Vto an image captured with the camera 61. FIG. 7 shows a visual field VRvisually recognized by the user through the image display section 20.

The user visually recognizes through the image display section 20 aperson, a building, a road, and other objects (hereinafter collectivelyreferred to as target object) present in a real world (hereinafterreferred to as outside scene) and a content displayed by the imagedisplay section 20. When the image display section 20 reproduces thecontent data 124 under the control of the control section 150, a contentis reproduced in the display region V. The display region V is a regionwhere the image display section 20 can display a content.

The display controlling section 155 causes the image display section 20to display a calibration image and recognizes and tracks a referenceactual object corresponding to the calibration image on the basis ofcaptured image data from the camera 61. In this state, the user movesthe position of the calibration image on the image display section 20via a user interface, such as the trackpad 14. The user then notifiesthe display controlling section 155 via the user interface at the timingwhen the user perceives that the calibration image is superimposed onthe reference actual object (at least one of position, size, andorientation of calibration image roughly coincides with that ofreference actual object). The display controlling section 155 produces,in response to the acquired notification, a parameter that relates aresult of the detection of the reference actual object (position incaptured image data) at the timing described above to the position wherethe calibration image is displayed in the display region V. The displaycontrolling section 155 causes the storage section 122 to store theproduced parameter.

The parameter is used, for example, when a content or a target object ina region pointed by the user with the hand/finger is specified.

The display controlling section 155 changes the display state of acontent displayed in the display region V by the image display section20 on the basis of the head motion (change in position of head) detectedby the six-axis sensor 235 and the magnetic sensor 237. The contentdisplayed in the display region V may, for example, be a document filecreated, for example, by Word or Excel or may be a motion image file ora still image file. The motion image file or the still image file may bea file stored in the storage section 122 in advance or may be a filedownloaded over a network under the control of the communication section117.

The coordinate setting section 157 sets virtual three-dimensionalspatial coordinates on the basis of the reference orientation set by thereference setting section 153. The coordinate setting section 157 in thepresent embodiment sets spherical coordinates, which are coordinates ina spherical coordinate system with the user's head located at the centerof the coordinate system. The spherical coordinates are expressed bythree-dimensional polar coordinates specified by the distance from theorigin, which is the user's head, and two angles of deviation. The twoangles of deviation are specified by an angle with respect to theorientation set as the reference orientation by the reference settingsection 153 and an angle with respect to the Z-axis direction, which isthe vertical direction.

A content used by the user via the display controlling section 155 isrelated to spherical coordinates set by the coordinate setting section157. When the user starts an application, the display controllingsection 155 relates a content displayed by the application tocoordinates in the spherical coordinate system. In a case where aplurality of contents are related to coordinates in the sphericalcoordinate system, the display controlling section 155 relates thecontents to coordinates in the spherical coordinate system in such a waythat relative display positions of the plurality of contents aremaintained. The coordinate setting section 157 then produces, as thedisplay position information 125, information for identification of theapplication that displays the content and information representing thesize of the display region where the content displayed by theapplication is displayed and the coordinates in the spherical coordinatesystem to which the content is related and causes the storage section122 to store the produced information.

The user can start a plurality of applications and cause a plurality ofcontents to be displayed in the display region V. In the case of the HMD100, however, when the number of contents displayed in the displayregion V increases, the visibility of the outside is hindered by thecontents in some cases. Although a plurality of contents are displayed,the user does not simultaneously operate the plurality of contents, andsome of the contents may not be temporarily used in some cases. In acase where a content that is not temporarily used is terminated for awhile and started again when the user desires to use the content, itundesirably takes time to start the application. It is thereforerequired not to display a content that is not temporarily used in thedisplay region V.

FIG. 8 shows a content related to coordinates in the sphericalcoordinate system.

When an application is started, the coordinate setting section 157relates a content displayed by the application to coordinates in thespherical coordinate system. FIG. 8 shows that a content is related tocoordinates in the spherical coordinate system that correspond to anangle of deviation ϕ with respect to the reference orientation, an angleof deviation θ with respect to the vertical direction, and a distance Rfrom the origin. In a case where the content is displayed in the form ofa rectangle, the coordinates related to the content may relate theangles of deviation θ and ϕ and the distance R to each of the fourcorners of the rectangle.

When motion of the head is detected by the head motion detecting section151, the display controlling section 155 moves the display region V,which represents the display range of the image display section 20, inthe spherical coordinate system in correspondence with the detectedhead's motion. In a case where the set coordinates of the content fallwithin the moved display region V, the display controlling section 155causes the image display section 20 to display the content.

FIGS. 9 and 10 are top views showing the user and contents set in thespherical coordinate system.

FIGS. 9 and 10 show a state in which a content A is set in the 0-degreedirection, which is the reference orientation direction, and a content Bis set in the direction rotated clockwise from the reference orientationby 45 degrees. Similarly, a content C is set in the direction rotatedclockwise from the reference orientation by 90 degrees, a content D isset in the 135-degree direction, and a content E is set in the180-degree direction. FIGS. 9 and 10 further show a state in which acontent F is set in the direction rotated clockwise from the referenceorientation by 225 degrees, a content G is set in the 270-degreedirection, and a content H is set in the 315-degree direction. Theangles 135 degrees, 180 degrees, and 225 degrees correspond to the rearof the user, and contents can be set on the rear of the user who facesin the reference orientation direction.

In a case where the user desires to visually recognize another contentin the spherical coordinate system, the user rotates the head in such away that the head faces in the direction where the content that the userdesires to visually recognize is set. The head motion detecting section151 detects the direction of the rotation of the head and the angle ofrotation of the head on the basis of detection values from the six-axissensor 235. The display controlling section 155 moves the display regionV in the spherical coordinate system in accordance with the direction ofthe rotation of the head and the angle of rotation of the head detectedby the head motion detecting section 151. In a case where thecoordinates of the set content are present in the moved display regionV, the display controlling section 155 causes the content to bedisplayed in the display region V. FIG. 9 shows a case where the userfaces in the 0-degree direction, which is the reference orientationdirection. In this case, since the coordinates of the content A arepresent in the display region V, the display controlling section 155causes the content A to be displayed in the display region V. FIG. 10shows a case where the user turns clockwise from the referenceorientation direction by 45 degrees. In this case, since the coordinatesof the content C are present in the display region V, the displaycontrolling section 155 causes the content C to be displayed in thedisplay region V.

FIG. 11 shows the user and contents set in the spherical coordinatesystem. In particular, FIG. 11 is a side view of the user.

A plurality of contents can be related to a single orientation angledirection in the spherical coordinate system. That is, contents can berelated to directions perpendicular to the direction of rotationdescribed above. Specifically, contents can be related to sphericalcoordinates by changing the angle of deviation θ with respect to thevertical direction.

FIG. 11 shows a content J related to coordinates where the angle ofdeviation θ is “α”, a content I related to coordinates where the angleof deviation θ is “β”, and a content K related to coordinates where theangle of deviation θ is “γ”. The coordinates α, β, and γ satisfy α<β<γ.

The user can visually recognize the content I in the display region V bycausing the head to face in the horizontal direction (direction in whichangle of deviation θ is 90 degrees) and can visually recognize thecontent J in the display region V by causing the head to face in anobliquely upward direction. The user can further visually recognize thecontent K in the display region V by causing the head to face in anobliquely downward direction.

FIG. 12 shows the user and contents set at coordinates in the sphericalcoordinate system.

Contents related to spherical coordinates may instead be set by changingthe distance R from the user. That is, contents may be disposed indifferent positions in the direction extending from the near side towardthe far side with respect to the user's head.

FIG. 12 shows a content S, which is separate from the origin by adistance R1 and related to coordinates where the angle of deviation θwith respect to the vertical direction is “β”, and a content T, which isseparate from the origin by a distance R2 and related to the coordinateswhere the angle of deviation θ with respect to the vertical direction is“β”.

When the user faces in the direction toward the spherical coordinateswhere the contents S and T are set, the contents S and T are displayedin the display region V. The user operates, for example, the trackpad14, the operation section 110, or the operation device 3 to select acontent to be operated or displayed. When the user selects one of thecontents, the display controlling section 155 changes the visibility ofthe contents in such a way that the visibility of the selected contentdiffers from the visibility of the non-selected content. For example,the display controlling section 155 increases the transparency of thenon-selected content as compared with the transparency of the selectedcontent or increases the luminance of the selected content as comparedwith the luminance of the non-selected content.

Further, the selected content may be clearly displayed by displaying atag in an end portion of each of the contents S and T and causing theportion of the tag in the selected content to blink or to be displayedin a different color.

In a case where the user faces in the direction toward coordinatesrelated to no content, a plurality of contents closest to thecoordinates may be displayed.

For example, assume a state in which a content is related to coordinateshaving the 0-degree deviation angle with respect to the referenceorientation and a content is related to coordinates having the 90-degreedeviation angle ϕ with respect to the reference orientation. In thisstate, when the user faces in the direction corresponding to the45-degree deviation angle ϕ, the content related to the 0-degreecoordinates and the content related to the 90-degree coordinates mayboth be displayed. That is, in a case where the user faces thecoordinates at the middle between the coordinates related to one of thecontents and the coordinates related to the other content, the contentsare both displayed.

The user desires in some cases to perform a task with a plurality ofcontents simultaneously displayed in the display region V. In suchcases, displaying two contents in the coordinates at the middle betweenthe coordinates related to the contents allows improvement in efficiencyat which the user performs the task.

The user can change a content and the coordinates in the sphericalcoordinate system that are related to the content. For example, thedisplay controlling section 155 may cause angles of deviation ϕ, such as45 degrees, 90 degrees, 135 degrees, and 180 degrees, and thumbnailimages of contents related to the angles of deviations ϕ to be displayedin an end portion of the display region V. The user may operate thetrackpad 14, the operation section 110, or the operation device 3, whilereferring to the displayed contents in the display region V, to changethe coordinates (angles of deviation ϕ and ϕ, distance R) in thespherical coordinate system that are related to each of the contents.

FIG. 13 shows that input operation is performed on the basis of thesightlines detected with the sightline sensors 68.

Having started an application and caused a content to be displayed inthe display region V, the user performs operation on the displayedcontent.

For example, in a case where the started application is browsersoftware, the display controlling section 155 may display a softwarekeyboard 170 as a content for input to allow the user to input a keywordin the search field in the browser screen displayed by the browsersoftware. The software keyboard 170 may be operated as follows: Thedirections of the user's sightlines are detected with the sightlinesensors 68; and a key corresponding to the sightline directions isdetermined to be the operated key on the software keyboard 170.

A key of the software keyboard 170 may instead be selected on the basisof an image captured with the camera 61.

The control section 150 causes the camera 61 in the image displaysection 20 to capture an image of the user's hand/finger and detects ahand/finger region where the hand/finger is seen on the basis of thecaptured image. The control section 150 identifies the shape of thehand/finger on the basis of the detected hand/finger region to identifya key of the software keyboard 170 or the key that overlaps with thefront end of the hand/finger. To identify a key of the software keyboard170 or the key that overlaps with the front end of the hand/finger, theparameter produced by the calibration is used. The control section 150uses the parameter to convert the coordinates of the hand/finger regionin the captured image into the coordinates in the display region V andidentifies a key of the software keyboard 170 or the key that overlapswith the front end of the hand/finger.

A key of the software keyboard 170 may instead be selected by operatinga cursor displayed in the display region V.

The cursor is moved over the software keyboard 170 in accordance withthe user's operation of the trackpad 14, the operation section 110, orthe operation device 3. The control section 150, when it acceptsfinalizing operation performed on the trackpad 14, the operation section110, or the operation device 3, identifies a key of the softwarekeyboard 170 or the key that corresponds to the position of the cursorat the time when the finalized operation is accepted and determines theselected key.

The cursor may instead be moved over the software keyboard in accordancewith the motion of the user's hand/finger or head.

In the case where the cursor is moved by the user's hand/finger, thecontrol section 150 detects a hand/finger region where the user'shand/finger is seen on the basis of an image captured with the camera61. The control section 150 uses, for example, a recognition dictionaryto identify the shape of the hand/finger on the basis of the detectedhand/finger region and moves the cursor in the direction pointed by thehand/finger. The recognition dictionary is information for identifyingthe shape and position of the hand/finger and is stored in the storagesection 122.

In the case where the cursor is moved by detected motion of the user'shead, the control section 150 moves the cursor on the basis of detectionvalues from the six-axis sensor 235. For example, in a case where theuser rotates the head clockwise, that is, in a case where rotation ofthe head around the Z axis is detected, the control section 150 movesthe cursor rightward by a predetermined distance. The same holds truefor a case where the user rotates the head counterclockwise. In a casewhere the user directs the head upward, the control section 150 detectsthe angle of rotation of the head around the X axis and moves the cursorupward in accordance with the detected angle of rotation. The same holdstrue for a case where the user directs the head downward.

When the cursor overlaps with a key of the software keyboard 170 thatthe user desires to select, the user performs finalizing operation offinalizing the key. The finalizing operation may be operation performedon the trackpad 14, the operation section 110, or the operation device3. In the case where the cursor is moved in accordance with theorientation of the user's hand/finger, the finalizing operation may beperformed by the motion of the user's head, and in the case where thecursor is moved in accordance with the motion of the user's head, thefinalizing operation may be performed by the user's hand/finger.

FIG. 14 shows an example of the content for input displayed at theuser's feet.

For example, the spherical coordinate system may be so set as tosurround the user (that is, the spherical coordinate system may be soset that the user is located at the center of the spherical coordinatesystem), and the content for input or any other content may be displayedat the user's feet. The control section 150 causes a content for inputto be displayed at the user's feet, and the content for input includesan image showing cursor movement directions corresponding to therightward and leftward directions and an image showing the finalizingoperation, as shown in FIG. 14. The control section 150 causes thecontent for input to be so displayed that the user is located at thecenter of the content for input.

The user instructs the HMD 100 in the cursor movement direction bystepping on the image showing the movement direction. Having moved thecursor, the user steps on the image showing the finalizing operation.The operation allows the HMD 100 to determine that a key that overlapswith the cursor has been selected.

When stepping on an image, the user directs the head downward so thatthe user's feet fall within the imaging range of the camera 61. Thecontrol section 150 evaluates the selected image on the basis of theimage captured with the camera 61. For example, in a case where a foothas stepped rightward, the control section 150 determines that the imagethat instructs the cursor to move rightward has been selected. In a casewhere a foot has stepped leftward, the control section 150 determinesthat the image that instructs the cursor to move leftward has beenselected. In a case where a foot has stepped forward to a point in frontof the body, the control section 150 determines that the finalizingoperation has been inputted.

The control section 150 may change the images displayed at the user'sfeet when the control section 150 accepts operation performed on thetrackpad 14, the operation section 110, or the operation device 3. Forexample, the control section 150 may cause a content for input to bedisplayed at the user's feet, and the content for input may includeimages showing cursor movement directions corresponding to the upwardand downward directions and the image showing the finalizing operation.Further, an image captured with the camera 325, which is incorporated inthe operation device 3, may be used to determine a selected image.

FIG. 15 shows an image for selecting an application or an applicationscreen. An application screen is a screen displayed by an application.

To select an application to be started from applications installed onthe HMD 100, the control section 150 may cause the image display section20 to display an image in which icons representing the applications arearranged in the form of a circle. Further, with the plurality ofapplications started, to select an application screen to be displayed inthe display region V, the control section 150 may cause the imagedisplay section 20 to display an image in which the application screensare arranged in the form of a circle. The control section 150 producesan image showing the application screens arranged in accordance withtheir coordinates related to spherical coordinate system and causes theimage display section 20 to display the image in the display region V.

To select an application or an application screen from the image inwhich the application representing icons or the application screens arearranged, the user may operate the trackpad 14, the operation section110, or the operation device 3 to select the application or theapplication screen or may select the application or the applicationscreen through operation performed by the user's foot.

To select an application or an application screen, the user causes thehead to face downward and steps forward with the right or left foot byone step. That is, the user allows the user's right or left foot to beimaged with the camera 61, which is incorporated in the image displaysection 20.

The control section 150 evaluates the foot with which the user hasstepped forward on the basis of the image captured with the camera 61.For example, in a case where the right foot is seen in the capturedimage, the control section 150 rotates clockwise the image in which theapplications or the application screens are displayed. In a case wherethe left foot is seen in the captured image, the control section 150rotates counterclockwise the image in which the applications or theapplication screens are displayed. In a case where an application or anapplication screen is stationary in a selection position shown in FIG.15 for at least a period set in advance, the control section 150determines that the application or the application screen has beenselected, starts the selected application or causes the selectedapplication screen to be displayed in the display region V.

FIGS. 16 and 17 are flowcharts showing the action of the HMD 100.

The control section 150 sets the reference state. To this end, thecontrol section 150 first evaluates whether or not it has accepted thefirst operation (step S1). In a case where a result of the evaluation isnegative (NO in step S1), the control section 150 waits until it acceptsthe first operation. In a case where a result of the evaluation isaffirmative (YES in step S1), the control section 150 evaluates whetheror not it has accepted the second operation (step S2). In a case where aresult of the evaluation is negative (NO in step S2), the controlsection 150 waits until it accepts the second operation. In a case wherea result of the evaluation is affirmative (YES in step S2), the controlsection 150 acquires detection values from the six-axis sensor and themagnetic sensor 237 (step S3).

The control section 150 then sets the axis of rotation of the head onthe basis of the detection values acquired from the six-axis sensor andthe magnetic sensor 237 and sets the reference orientation on the basisof the axis of rotation of the head and the position of the head havingrotated around the axis of rotation (step S4).

The control section 150 then evaluates whether or not it has acceptedoperation that starts an application (step S5). In a case where a resultof the evaluation is negative (NO in step S5), the control section 150proceeds to the evaluation in step S10. The evaluation in step S10 willbe described later. In a case where a result of the evaluation isaffirmative (YES in step S5), the control section 150 evaluates whetheror not any other application had been started (step S6).

In a case where a result of the evaluation in step S6 is negative (NO instep S6), the control section 150 relates the screen of the startedapplication (hereinafter referred to as application screen) to thecoordinates in the spherical coordinate system that correspond to thedirection of the reference orientation set in step S4 (step S8). In acase where a result of the evaluation in step S6 is affirmative (NO instep S6), the control section 150 relates the screens of theapplications that had been started to other coordinates in the sphericalcoordinate system (step S7). The control section 150 then relates thestarted application screen to the coordinates in the sphericalcoordinate system that correspond to the direction of the referenceorientation set in step S4 (step S8). The control section 150 thencauses the image display section 20 to display the started applicationscreen in the display region V (step S9).

The processes in steps S5 to S9 will be described in more detail.

The description will be made on the assumption that the user's headfaces in the reference orientation direction set in the reference statesetting action.

For example, when the user starts an application, the displaycontrolling section 155 causes the image display section 20 to displaythe screen displayed by the application (hereinafter referred to asapplication screen A). The display controlling section 155 furtherrelates the application screen A to coordinates in the sphericalcoordinate system. Specifically, the display controlling section 155relates the application screen A to the coordinates in the sphericalcoordinate system where the angle of deviation ϕ is 0 degrees. In thefollowing description, it is assumed that the distance R to which anapplication screen is related and the angle of deviation ϕ with respectto the vertical direction to which the application screen is related areset at the same values, and no description of the distance R and theangle of deviation ϕ will be made.

With the application screen A displayed in the display region V, whenthe user starts a next application (hereinafter referred to asapplication B), the display controlling section 155 changes thecoordinates of the application screen A in the spherical coordinatesystem.

The display controlling section 155 determines that the probability ofselection of the screen of the started application B (hereinafterreferred to as application screen B) as a target to be used or displayedis higher than the probability of selection of the application screen A.The display controlling section 155 therefore, for example, relates theapplication screen A to the coordinates where the angle of deviation ϕis 90 degrees and relates the application screen B to the position wherethe angle of deviation ϕ is 0 degrees. Since the user's head faces inthe direction corresponding to the angle of deviation ϕ of 0 degrees,the application screen B is displayed in the display region V.

Thereafter, with the application screen B displayed in the displayregion V, when the user starts a next application (hereinafter referredto as application C), the display controlling section 155 changes thecoordinates of the application screens A and B in the sphericalcoordinate system.

Also in this case, the display controlling section 155 determines thatthe probability of selection of the screen of the started application C(hereinafter referred to as application screen C) as a target to be usedor displayed is higher than the probability of selection of theapplication screen A or B. The display controlling section 155therefore, for example, relates the application screen A to thecoordinates where the angle of deviation ϕ is 180 degrees and relatesthe application screen B to the coordinates where the angle of deviationϕ is 90 degrees. The display controlling section 155 then relates theapplication screen C to the position where the angle of deviation ϕ is 0degrees. Since the user's head faces in the direction corresponding tothe angle of deviation ϕ of 0 degrees, the application screen C isdisplayed in the display region V.

In a case where the user desires to display the application screen A ofthe application A during operation of the application screen C, the userrotates the head by 180 degrees. The head motion detecting section 151detects the direction of the rotation of the head and the angle of therotation thereof on the basis of detection values from the six-axissensor 235, and the display controlling section 155 moves the positionof the display region V in the spherical coordinate system in accordancewith the direction of the rotation of the head and the angle of therotation thereof detected by the head motion detecting section 151. In acase where the coordinates of the set content are present in the moveddisplay region V, the display controlling section 155 causes the contentto be displayed in the display region V. The user who faces in thedirection corresponding to the angle of deviation ϕ of 180 degreestherefore allows the application screen A to be displayed in the displayregion V.

The description of the action of the HMD 100 will be resumed withreference to the flowchart shown in FIG. 17.

In the case where an application screen is displayed in the displayregion V in step S9, or in the case where a result of the evaluation instep S5 is negative, the control section 150 evaluates whether or notany head motion has been detected on the basis of detection values fromthe six-axis sensor 235 and the magnetic sensor 237 (step S10). In acase where a result of the evaluation is negative (NO in step S10), thecontrol section 150 returns to the evaluation in step S5 and evaluateswhether or not an application has been started (step S5).

In a case where a result of the evaluation in step S10 is affirmative(YES in step S10), the control section 150 evaluates whether or not anyother application has been already started (step S11). In a case where aresult of the evaluation is negative (NO in step S11), the controlsection 150 proceeds to the evaluation in step S13. In step S13, thecontrol section 150 evaluates, on the basis of detection values from thesix-axis sensor 235 and the magnetic sensor 237, whether or not thereference orientation needs to be set again. That is, in some cases, theuser unintentionally moves the head and the direction of the setreference orientation is undesirably shifted. In a case where a resultof the evaluation shows that the user's head has moved from thedirection set as the reference orientation direction by at least apredetermined value, the control section 150 determines that thereference orientation needs to be set again. In a case where the user'shead has not moved from the direction set as the reference orientationdirection by at least the predetermined value, the control section 150determines that the reference orientation does not need to be set again.

In a case where a result of the evaluation in step S13 is negative (NOin step S13), the control section 150 returns to the evaluation in stepS5 and evaluates whether or not an application has been newly started.In a case where a result of the evaluation in step S13 is affirmative(YES in step S13), the control section 150 causes the image displaysection 20 to display a message stating “Reference orientation is setagain” and proceeds to the evaluation in step S2.

In a case where a result of the evaluation in step S11 is affirmative(YES in step S11), that is, in a case where an application has beenstarted, the control section 150 detects the direction of the rotationof the head and the angle of the rotation thereof on the basis ofdetection values from the six-axis sensor 235 and the magnetic sensor237. The control section 150 then moves the position of the displayregion V in the spherical coordinate system in accordance with thedetected direction of the rotation of the head and the detected angle ofthe rotation thereof. Thereafter, in a case where the coordinates of theset application screen are present in the moved display region V, thecontrol section 150 causes the application screen to be displayed in thedisplay region V (step S12).

The control section 150 then evaluates whether or not it has acceptedapplication terminating operation (step S14). In a case where a resultof the evaluation is negative (NO in step S14), the control section 150returns to the evaluation in step S5 and evaluates whether or not anapplication has been newly started. In a case where a result of theevaluation in step S14 is affirmative (YES in step S14), the controlsection 150 terminates the application for which the control section 150has accepted the terminating operation and cancels the relationshipbetween the application screen displayed by the application and thecoordinates in the spherical coordinate system. The control section 150then evaluates whether or not every started applications has beenterminated (step S15). In a case where a result of the evaluation isnegative (NO in step S15), the control section 150 returns to theevaluation in step S5 and evaluates whether or not an application hasbeen newly started. In a case where a result of the evaluation in stepS15 is affirmative (YES in step S15), the control section 150 terminatesthe process procedure.

In the embodiment described above, the reference state is set on thebasis of the position or motion of the head. Instead, the referencestate may be set on the basis of the position or motion of the user'shand/finger or arm as the pointing element. For example the user maywear the operation device 3, in which the six-axis sensor and themagnetic sensor are incorporated, and the reference state may be set onthe basis of detection values from the sensors in the operation device3. Specifically, the user extends the arm on which the operation device3 is worn in a direction that the user desires to set as the referenceorientation. The operation device 3 transmits detection values from thesix-axis sensor and the magnetic sensor to the HMD 100. The HMD 100 setsthe reference orientation on the basis of the detection valuestransmitted from the operation device 3.

In the aforementioned description, the display region V is moved in thespherical coordinate system and a content to be displayed is determined.Instead, the position of the display region V may be fixed, and thespherical coordinate system is rotated in accordance with motion of thehead. After the spherical coordinate system is rotated, a contentlocated in the display region V may be identified.

To rotate spherical coordinate system, it may be rotated around the axisof rotation set through the head. The axis of rotation may be so set asto be perpendicular to the ground or may be set on the basis of thedisplay surface of the operation device 3 worn by the user and the axisof rotation of the head. The axis of rotation around which the sphericalcoordinate system is rotated may be changed in accordance with a sceneto be used.

The display magnification of a content displayed in the display region Vmay be changed on the basis of the position and motion of the user'shand/finger.

For example, a gesture dictionary for identifying a gesture based on themotion or position of the user's hand/finger is stored in the storagesection 122 in advance. The HMD 100 detects a hand/finger region inwhich the user's hand/finger is seen on the basis of an image capturedwith the camera 61 and detects the motion, shape, and position of thehand/finger. The HMD 100 compares the detected motion, shape, andposition of the hand/finger with information for identifying gesturesregistered in the gesture dictionary to identify the detected motion asa gesture. The HMD 100 carries out a process corresponding to theidentified gesture. For example, in a case where the identified gestureis a gesture related to operation of increasing the displaymagnification, the HMD 100 changes the display magnification of thecontent displayed in the display region V and causes the content to beenlarged and displayed.

The aforementioned embodiment has been described with reference to thecase where spherical coordinates are used as the coordinates set by thecoordinate setting section 157, but the coordinate setting section 157may set a columnar coordinate system located at a predetermined distancefrom the origin or the axis of rotation of the user's head. Thecoordinates set by the coordinate setting section 157 may be coordinatesset in a plane located at a predetermined distance from the origin orthe axis of rotation of the user's head. Further, the aforementionedembodiment has been described with reference to the spherical coordinatesystem that is rotational around the axis of rotation passing throughthe origin where the user is located, but the position of the origin maynot coincide with the position of the axis of rotation.

In the embodiment described above, the position and motion of the headare detected on the basis of detection values from the six-axis sensor235 and the magnetic sensor 237, and the display region V showing thedisplay range of the image display section 20 is moved in the sphericalcoordinate system in correspondence with the detected motion of thehead. As another method, the storage section 122 may store informationon the history of the directions in which the user directed the head inthe past, and the motion of the head may be predicted on the basis ofthe history information. The display controlling section 155 may selecta content in accordance with a result of the prediction and display theselected content in the display region V.

As described above, the HMD 100 according to the present embodimentincludes the image display section 20, the six-axis sensor 235, themagnetic sensor 237, and the head motion detecting section 151 as thedetection section, the reference setting section 153, and the displaycontrolling section 155.

The six-axis sensor 235, the magnetic sensor 237, and the head motiondetecting section 151 as the detection section detect at least one ofthe position and motion of the head.

The reference setting section 153 sets the reference state on the basisof at least one of the head position and motion detected by thedetection section. The display controlling section 155 changes thedisplay state of a content displayed by the image display section 20 onthe basis of changes in the head position and motion with respect tothose in the reference state.

Further, the display controlling section 155 changes the display statesof a plurality of contents displayed by the display section in such away that the relative display positions of the contents are maintained.

Therefore, a content to be displayed can be readily selected, and thedisplay state of the content can be changed through simple operation.

The HMD 100 further includes the storage section 122, which stores thedisplay position information representing relative display positions ofthe plurality of contents.

The display controlling section 155 changes the display states of theplurality of contents in such a way that the relative display positionsof the contents represented by the display position information aremaintained. The display states of the contents displayed by the imagedisplay section 20 can therefore be so changed that the relative displaypositions of the plurality of contents are maintained.

The display controlling section 155 selects a content on the basis ofchanges in the position and motion of the head with respect to those inthe reference state and causes the image display section 20 to displaythe selected content.

Therefore, a content can be selected on the basis of the position ormotion of the head, and the selected content can be displayed by theimage display section 20.

The reference setting section 153 sets, as the reference state, the axisof rotation of the head and the position of the head that rotates aroundthe axis of rotation.

The HMD 100 further includes the coordinate setting section 157. Thecoordinate setting section 157 can set the display position of contentsalong the rotational direction around the axis of rotation and along thedirection perpendicular to the rotational direction.

A plurality of contents can therefore be so set that the relativedisplay positions of the plurality of contents are maintained.

The coordinate setting section 157 can set the display position of acontent on the rear side of the head in the reference state. The displayposition of a content can therefore be changed in a variety of manners.

In the embodiment of the invention, the display controlling section canset the display position of a content on the rear side of the detectiontarget section in the reference state.

According to the embodiment of the invention, the display position of acontent can be changed in a variety of manners.

The coordinate setting section 157 can set the display positions of aplurality of contents in positions where the user can visually recognizethe contents arranged on a spherical surface around a point on the axisof rotation or in a plane separate from the axis of rotation by apredetermined distance.

The display states of the plurality of contents can therefore be sochanged that the relative display positions of the contents aremaintained.

The display controlling section 155 can set the display position of acontent in a position where the user can visually recognize the contentdisposed in the plane containing the user's feet.

The user is therefore allowed to visually recognize the content disposedin the plane containing the user's feet.

The image display section 20 includes the right display unit 22, whichallows the user's right eye to visually recognize an image and the leftdisplay unit 24, which allows the user's left eye to visually recognizean image.

The display controlling section 155 causes a content to be displayedwith parallax present between the right display unit 22 and the leftdisplay unit 24.

The content can therefore be displayed in a three-dimensional manner.

The reference setting section 153 sets the reference state in accordancewith the position or motion of the pointing element. The reference statecan therefore be readily set.

The display controlling section 155 sets the display magnification of acontent in the image display section 20 in accordance with the positionor motion of the pointing element.

The display magnification of a content can therefore be set throughsimple operation.

The display controlling section 155 causes the image display section 20to display the content for input used for input operation performed onany of the contents.

Input operation can therefore be performed on a content via the contentfor input.

The reference setting section 153 sets, when it determines that the headhas been stationary for at least a predetermined period, the position ofthe stationary head as the reference state.

The reference state will therefore not be set in a position where theuser does not intend.

The HMD 100 further includes the sightline sensors 68, which detect theorientations of the user's sightlines. The reference setting section 153sets the reference state including the position or motion of the headand the orientations of the sightlines detected with the sightlinesensors 68.

The reference state can therefore be set in more detail incorrespondence with the user's state.

The detection section is disposed in the enclosure of the image displaysection 20 and includes at least any of a GPS detection section thatperforms position detection on the basis of GPS signals, an accelerationsensor that detects acceleration, an angular acceleration sensor thatdetects angular acceleration, a monocular camera, a stereocamera, and amagnetic sensor.

The detection section can therefore detect the position or motion of thehead with precision.

The magnetic sensor 237 is disposed on the side facing the outer surfaceof the enclosure of the image display section 20.

The influence of the enclosure on the magnetism detection performed bythe magnetic sensor 237 can therefore be reduced.

The HMD 100 further includes the control section 150, which performs aplurality of functions, and each of the contents corresponds to thefunctions performed by the control section 150.

A content corresponding to the functions performed by the controlsection 150 can therefore be displayed by the image display section 20.

The control section 150 functions as the editing processing section thatchanges the substance of a content displayed by the image displaysection 20. The control section 150 can therefore change the substanceof a content.

The display controlling section 155 causes a plurality of contentsdisplayed by the image display section 20 to be displayed in the form ofthumbnail images on the basis of changes in the position and motion ofthe head with respect to those in the reference state.

The user can therefore readily grasp the contents displayed by the imagedisplay section 20.

The display controlling section 155 predicts changes in the position andmotion of the head with respect to those in the reference state andchanges the display state of a content in accordance with a result ofthe prediction.

The display state of a content can therefore be changed at an earlytiming.

The display controlling section 155 causes the image display section 20to display a plurality of contents in such a way that they are arrangedin different positions in the direction extending from the near sidetoward the far side with respect to the head.

Contents displayed by the image display section 20 can therefore bedisplayed in a plurality of positions.

The invention is not limited to the configuration of the embodimentdescribed above and can be implemented in a variety of aspects to theextent that the aspects do not depart from the substance of theinvention.

For example, the aforementioned embodiment has been described withreference to the case where the control apparatus 10 is wired to theimage display section 20, but not necessarily in the invention, and theimage display section 20 may be wirelessly connected to the controlapparatus 10. The wireless communication method in this case may be anyof the methods exemplified as the communication methods handled by thecommunication section 117 or any other communication method.

In the embodiment described above, the position and motion of the headare detected by using the six-axis sensor 235 and the magnetic sensor237. The position and motion of the head may be detected on the basis ofthe following information in addition to the sensors.

The position and motion of the head may be detected on the basis, forexample, of the electric wave strength of GPS signals received by theGPS receiver 115, the electric wave strength of a beacon signaltransmitted from a beacon signal transmitter (not shown), an imagecaptured with the camera 61, and a detection value from a vibrationdetecting sensor. The position and motion of the head may be detected onthe basis of the combination of detection values from the six-axissensor 235, the magnetic sensor 237, and the vibration detection sensor,the electric wave strengths of the GPS signals and the beacon signal,the captured image, and other pieces of information.

The HMD 100 may include an interface (not shown) that connects a varietyof external apparatus that serve as content supply sources. Theinterface may, for example, be an interface that supports wiredconnection, such as a USB interface, a micro-USB interface, and a memorycard interface, and may instead be formed of a wireless communicationinterface. The external apparatus in this case are each an image supplyapparatus that supplies the HMD 100 with images and are, for example, apersonal computer (PC), a mobile phone terminal, and a mobile gameconsole. In this case, the HMD 100 can output images and voice based oncontent data inputted from the external apparatus.

Part of the functions provided by the control apparatus 10 may beprovided by the image display section 20, or the control apparatus 10may be achieved by a plurality of apparatus. That is, the controlapparatus 10 does not necessarily have the configuration including thebox-shaped case 10A. For example, in place of the control apparatus 10,a wearable device that can be attached onto the user's body, the user'sclothing, or an accessory worn by the user may be used. The wearabledevice in this case may, for example, be a timepiece-shaped apparatus, afinger-ring-shaped apparatus, a laser pointer, a mouse, an air mouse, agame controller, or a pen-shaped device.

Further, in the embodiment described above, the description has beenmade of the case where the image display section 20 is separated fromthe control apparatus 10 and they are connected to each other via theconnection section 40, but not necessarily in the invention. The controlapparatus 10 and the image display section 20 can instead be integratedwith each other, and the integrated unit can be worn on the user's head.

The control apparatus 10 may be a notebook computer, a tablet computer,or a desktop computer. Instead, the control apparatus 10 may, forexample, be a mobile electronic apparatus including a game console, amobile phone, a smartphone, and a portable media player, or any otherdedicated apparatus.

In the embodiment described above, the configuration in which the userviews an outside scene through the display section is not limited to theconfiguration in which the right light guide plate 26 and the left lightguide plate 28 transmit outside light. For example, the invention isalso applicable to a display apparatus that displays an image but doesnot allow the user to visually recognize an outside scene. Specifically,the invention is applicable to a display apparatus that displays animage captured with the camera 61, an image and a CG produced on thebasis of the captured image, video images based on prestored video dataor externally inputted video data, and other images. An example of adisplay apparatus of this type may include a display apparatus that doesnot allow the user to visually recognize an outside scene or what iscalled a closed-type display apparatus. Further, the invention isapplicable to a display apparatus that does not perform the displaydescribed above, that is, AR display, in which an image is so displayedas to be superimposed on the real space, or MR (mixed reality) display,in which a captured real space image is combined with an imaginaryimage. The invention is also applicable to a display apparatus that doesnot perform VR (virtual reality) display, in which a virtual image isdisplayed. Further, for example, a display apparatus that displaysexternally inputted video data or an analog video signal is, of course,an apparatus to which the invention is applied.

Further, for example, the image display section 20 may be replaced withan image display section worn, for example, as a cap or any other imagedisplay section worn based on a different method. A display section thatdisplays an image in correspondence with the user's left eye LE and adisplay section that displays an image in correspondence with the user'sright eye RE only need to be provided. Moreover, the display apparatusaccording to the embodiment of the invention may, for example, beconfigured as a head mounted display incorporated in an automobile, anairplane, and other vehicles. Further, for example, the displayapparatus may be configured as a head mounted display built in a helmetor any other body protection gear. In this case, a portion thatpositions the display section relative to the user's body and a portionthat is positioned relative to the positioning portion can be a portionworn by the user.

The configuration in which the half-silvered mirrors 261 and 281 formvirtual images in part of the right light guide plate 26 and the leftlight guide plate 28 is presented as an example of the optical systemthat guides the image light to the user's eyes. The configurationdescribed above does not necessarily employed in the invention, and aconfiguration in which an image is displayed in a display region havingan area that occupies the entire or majority of each of the right lightguide plate 26 and the left light guide plate 28 may be employed. Inthis case, the action of changing the position where an image isdisplayed may include the process of reducing the size of the image.

Further, the optical elements in the embodiment of the invention are notlimited to the right light guide plate 26 and the left light guide plate28 having the half-silvered mirrors 261 and 281 and only need to beoptical parts that cause the image light to be incident on the user'seyes. Specifically, a diffraction grating, a prism, or a holographydisplay section may be used.

At least part of the functional blocks shown in FIGS. 4 and 5 and otherfigures may be achieved by hardware or cooperation between hardware andsoftware, and the configuration formed of the independent hardwareresources shown in FIGS. 4 and 5 and other figures is not necessarilyemployed. Further, the program executed by the control section 150 maybe stored in the nonvolatile storage section 121 or another storagedevice (not shown) in the control apparatus 10, or a program stored inan external apparatus may be acquired via the communication section 117or the external connector 184 and executed. Among the configurationsformed in the control apparatus 10, the operation section 110 may beformed as a user interface (UI).

The process units in the flowcharts shown in FIGS. 16 and 17 are thosedivided in accordance with primary processing contents for ease ofunderstanding of the entire process carried out by the control section10 of the HMD 100, and the invention is not limited by how to producethe divided process units or the name thereof. In accordance with theprocess contents, the entire process carried out by the control section150 may be divided into a greater number of process units, or oneprocess unit may further be divided into a large number of processes.The order in which the process units in the flowcharts described aboveare carried out is not limited to the order in the illustrated example.

The entire disclosure of Japanese Patent Application No. 2016-245704,filed Dec. 19, 2016 is expressly incorporated by reference herein.

What is claimed is:
 1. A display apparatus comprising: a display sectionthat is worn on a user's head; a detection section that detects at leastone of a position and a motion of a detection target section includingthe head; a reference setting section that sets a reference state basedon at least one of the position and motion of the detection targetsection detected by the detection section; and a display controllingsection that changes display states of contents displayed by the displaysection based on changes in the position and motion of the detectiontarget section with respect to the position and motion of the detectiontarget section in the reference state, wherein the display controllingsection changes the display states of the contents displayed by thedisplay section in such a way that relative display positions of aplurality of the contents are maintained.
 2. The display apparatusaccording to claim 1, further comprising a storage section that storesdisplay position information representing the relative display positionsof a plurality of the contents, wherein the display controlling sectionchanges the display states of the contents in such a way that therelative display positions of a plurality of the contents represented bythe display position information are maintained.
 3. The displayapparatus according to claim 1, wherein the display controlling sectionselects one of the contents based on changes in the position and motionof the detection target section with respect to the position and motionof the detection target section in the reference state and causes thedisplay section to display the selected content.
 4. The displayapparatus according to claim 1, wherein the reference setting sectionsets the reference state based on an axis of rotation of the detectiontarget section and the position of the detection target section thatrotates around the axis of rotation, and the display controlling sectionis capable of setting the display positions of the contents along arotational direction around the axis of rotation and along a directionperpendicular to the rotational direction.
 5. The display apparatusaccording to claim 4, wherein the display controlling section is capableof setting the display position of any of the contents on a rear side ofthe detection target section in the reference state.
 6. The displayapparatus according to claim 4, wherein the display controlling sectionis capable of setting the display positions of the contents in positionswhere the user is allowed to visually recognize a plurality of thecontents arranged on a spherical surface around a point on the axis ofrotation or in a plane separate from the axis of rotation by apredetermined distance.
 7. The display apparatus according to claim 4,wherein the display controlling section is capable of setting thedisplay position of any of the contents in a position where the user isallowed to visually recognize the content disposed in a plane containingthe user's feet.
 8. The display apparatus according to claim 1, whereinthe display section includes a right display section that allows theuser's right eye to visually recognize an image and a left displaysection that allows the user's left eye to visually recognize an image,and the display controlling section causes any of the contents to bedisplayed with parallax present between the right display section andthe left display section.
 9. The display apparatus according to claim 1,wherein the reference setting section sets the reference state inaccordance with a position or a motion of a pointing element.
 10. Thedisplay apparatus according to claim 9, wherein the display controllingsection sets a display magnification of any of the contents in thedisplay section in accordance with the position or motion of thepointing element.
 11. The display apparatus according to claim 1,wherein the display controlling section causes the display section todisplay a content for input used for input operation performed on any ofthe contents.
 12. The display apparatus according to claim 1, whereinthe reference setting section sets, when the reference setting sectiondetermines that the detection target section has been stationary for atleast a predetermined period, the position of the stationary detectiontarget section as the reference state.
 13. The display apparatusaccording to claim 1, further comprising a sightline detection sectionthat detects orientations of the user's sightlines, wherein thereference setting section sets the reference state including theposition or motion of the detection target section and the orientationsof the sightlines detected by the sightline detection section.
 14. Thedisplay apparatus according to claim 1, wherein the detection section isdisposed in an enclosure of the display section and includes at leastany of a GPS detection section that performs position detection based onGPS signals, an acceleration sensor that detects acceleration, anangular acceleration sensor that detects angular acceleration, amonocular camera, a stereocamera, and a magnetic sensor.
 15. The displayapparatus according to claim 14, wherein the magnetic sensor is disposedon a side facing an outer surface of the enclosure of the displaysection.
 16. The display apparatus according to claim 1, furthercomprising a processing section that performs a plurality of functions,wherein each of the contents corresponds to the functions performed bythe processing section.
 17. The display apparatus according to claim 1,further comprising an editing processing section that changes asubstance of any of the contents displayed by the display section. 18.The display apparatus according to claim 1, wherein the displaycontrolling section causes a plurality of the contents displayed by thedisplay section to be displayed in a form of thumbnail images based onchanges in the position and motion of the detection target section withrespect to the position and motion of the detection target section inthe reference state.
 19. The display apparatus according to claim 1,wherein the display controlling section predicts changes in the positionand motion of the detection target section with respect to the positionand motion of the detection target section in the reference state andchanges the display state of any of the contents in accordance with aresult of the prediction.
 20. The display apparatus according to claim1, wherein the display controlling section causes the display section todisplay a plurality of the contents in such a way that the plurality ofcontents are arranged in different positions in a direction extendingfrom a near side toward a far side with respect to the detection targetsection.
 21. A method for controlling a display apparatus including adisplay section that is worn on a user's head, the method comprising: adetecting step of detecting at least one of a position and a motion of adetection target section including the head; a setting step of setting areference state based on at least one of the position and motion of thedetection target section detected in the detecting step; and a changingstep of changing display states of contents displayed by the displaysection based on changes in the position and motion of the detectiontarget section with respect to the position and motion of the detectiontarget section in the reference state, wherein the display states of thecontents displayed by the display section are so changed that relativedisplay positions of a plurality of the contents are maintained.